VALIDATION TESTS | LEKKER UPDATE #8

I have to start with an apology here. Since the last update, I have not updated the project timeline on the update page. I realize this created a bit of confusion about the expectation. Especially if you don't read the updates in detail.

There are three reasons why I was hesitant to update it:

• We had no firm confirmation dates from the manufacturer yet.
• We were waiting for test results that would decide if things would take longer or not.
• Strongly dislike adjusting that schedule frequently.

In this update, I’ll publish a new schedule and fill in exact dates every time the manufacturer confirmed. Otherwise, I’m estimating the month it will take place based on experience and feedback from the manufacturer. You can find the new schedule under the Timeline header.

Let’s get into the body of the update, there’s apparently a lot to share.

The Circuit Board (PCBA)

In the DVT phase, the Wooting two Lekker Edition (LE) undergoes certification, conformance, and operational tests (saltwater, humidity, temperatures, etc.). Certification also includes CE, FCC, WEEE, and RoHs. To get CE or FCC certified there’s one all-important factor: Electro-Magnetic Interference (EMI) emittance.

All electronic devices emit a small electromagnetic field that can influence other electronic devices. The goal of the EMI (within the FCC/CE certification) test is to ensure that your device doesn’t interfere with other devices within a certain proximity. The problem is that EMI is a complicated mess of reasons that are not easily broken down into linear issues. This is why it’s very common for electronic devices to fail an EMI test 3-5 times before it’s approved. The result of these tests can’t tell you why or what is wrong. Instead, it’ll give you a list of emittance frequencies that are too high and could affect other devices. Fix it.

To avoid failing CE/FCC certification due to EMI issues, we went through a pre-EMI test. The manufacturer simulates the FCC/CE certificate EMI test to find out beforehand if there’s any issue. We failed the first attempt by a hair, a setback. Back to the drawing board.

There’s a lot of good practice you can follow while developing a circuit board to reduce EMI. This significantly helps reduce the chances of failing the test, but there’s no guarantee. Remember when I mentioned that the PCBA changes seemed to never end and always felty nearly done? Jeroen was making these exact types of changes based on the manufacturer’s feedback. Every step closer they got to the final design, they were able to find another possible improvement.

This might explain why we failed the test by a hair on the first try. Since there’s no clear explanation as to why you failed the test, unless you go through a lot more trial and error, the common approach is to just shotgun it. This is an aggressive approach by adding components in different areas that could possible contribute to EMI, in the hopes that one of them or the combination of them hits the causation. It’s especially effective if you don’t have another 3 months to figure out the exact reason, it’s complicated.

In our case, we didn’t have to go overboard. We added a few components in a few areas, tweaked the software, and ran it for another test. It passed. But it also meant we had to make a new drawing and another run of new PCBAs for the official certification test. If we didn’t pass this test the second time, it would have easily delayed everything by a month. You can say we dodged a bullet here.

Bottom case mold

Our new assembly manufacturer wanted to add a few more adjustments to the bottom case mold to improve conformity with their parts and ensure a smoother assembly. This would be a speedy and painless procedure.

However, when they received the Wooting two mold and ran a test injection, they were able to find a few serious issues. There was a defect in one of the cooling systems and noticed they could improve the quality by adding gas-assisted injection. This would help lessen the tolerances, defects, and create more consistency. This is all great, but it also meant it needed more time and we had to spend even more unexpected costs 💵..

They weren’t able to start on this until the 15th of November. All the changes should finish by the 29th of November. Relatively fast.

This also means that the original October DVT timeline got killed. The PCBA would’ve delayed this to November, but now also with the added mold adjustments, it was pushed to December. Thankfully, we were able to get a hard confirmation that DVT will take place on the 17th of December. There’s very little reason for this date to change, as everything is settled.

The top plate

In the last update, I shared a few pictures showing that the EVT top plate color too dark. I still had to double-check if it matched with the signed color plates. There are in total 4 signed plates with copies for different parties.

3 plates are for our assembly manufacturer’s incoming quality control and reference for our supplier. The 3 plates are divided into an upper, middle, and bottom limit color variance. This is done to set a color tolerance for visual inspection. It’s not perfect, but it’s extremely difficult to keep consistency in anodization colors.

The other 1 plate was the original plate that Kevin signed with the original supplier as a reference. This was before the 3 quality control plates were made. I discovered that none of the signed plates precisely matched the EVT sample top plate. I also figured that the 3 quality control plates were too dark and missing a lot of the signature blues. In contrast to the original signed plate, which struck the right balance.

Why these 3 quality control plates didn’t exactly match the original plate has a mixture of reasons, I’d throw it mostly to time pressure and unfamiliar process with new assembly manufacturer.

Kevin discussed this issue with the top plate manufacturer, asked him to do a better job, and send updated color samples together with the Wooting two HE color samples (😄). I received those a few days ago, and they are perfect.

There was however one issue, this color is way too different from the 3 quality control plates. If the supplier would deliver these top plates, the assembly manufacturer would outright reject them, and send them back. The last thing I wanted was to repeat the whole approval process again. This bureaucracy would set us back by 2-3 weeks. I talked it over with the manufacturer and we came to a good solution.

Instead of using the 3 plates with the upper, middle, and lower limit for incoming quality control. We’re going to use these new plates as a reference for a spectrophotometer. Apparently, this is a device that can analyze color and produce a numerical value. Based on this numerical value we can add a tolerance (delta) to cover the upper and lower color variation limits. I’m a little dumbfound why this is not a standard procedure, instead of visual inspection of samples. On the other hand, I heard that it’s reliable enough for anodized colors and I can imagine that different brand machines will produce different results.

In either case, it solves the issue and now we have our Lekker top plate!

Minor part: The puller

Minor part, major topic.

We couldn’t use the orange double-sided switch/keycap puller from our Wooting one and two anymore. The switch puller didn’t work for the Lekker switches, due to its tooth design, rendering it useless. I had to look for an alternative switch and keycap puller but it was utterly disappointing to find out what the options were.

My first thought was to use an IC puller for switches and a wire puller for keycaps, you might recognize these.

The IC puller is often used as a switch puller by the keyboard community. They’re not amazing, but get the job done. I thought there would be a wide availability for these, but I was wrong. There was a high minimum order quantity and no color options, just green. If we wanted to modify the color, the MOQ would jump to 50 000 pieces. It was also not RoHs certified. All our parts need to be RoHS certified, otherwise, we need to do it for them.

I suspect since we work with a fairly large assembly manufacturer that any suppliers outside their usual network will quote stupid numbers. It was really not working in our favor, and this wasn’t one of the parts I wanted to source on consignment.

There was an alternative, but it looks horrible. I refer to it as the CIY puller, it’s been around for a while and it works as a double-sided puller.

CIY stands for “Change It Yourself”, it is a term made up by a Chinese manufacturer for the first commercially available hot-swappable switch keyboards. It’s probably better known in China. It’s only recently that the hot-swap technology for traditional switches has become reliable enough to be widely adopted.

Anyway, it gets the job done and for all I cared it would fill the void until we found a better alternative…. that was until I got quoted on these things. I can’t disclose prices here, but it was priced 4 times the wire keycap puller. And here’s the kicker, the MOQ was 5000 pieces. If we wanted to laser our logo on the surface the MOQ would bump up to 10 000 pieces. It sounded utterly ridiculous to me.

If you’re a manufacturer that offers an ODM product with the option to brand, the whole point and benefit should be lower MOQ and custom brand at a reasonable per piece price. An ODM product comes with the risk of competitors being able to purchase the exact same product with their own logo on it. It’s not unique.

To make it even more interesting, I got a new quotation through a trading company on Alibaba instead of our assembly manufacturer. I was able to get the MOQ to 1000 pieces with a custom brand at 50% of the cost. It confirmed my previously stated suspicion. This was grinding my gears.

I decided that we should just drop the puller entirely from the Wooting two Lekker Edition/HE, only offer the wire keycap puller and start development on our own switch puller. I didn’t want to risk relying on any trading company for a core product. If they screw up, they jeopardize the entire product, and you can’t rely on them to deliver as promised since they’re not the original manufacturer.

It’s not ideal, but realistically speaking, even if we added a switch puller, you won’t need it, you won’t use it. There’s only 1 switch type at this moment (Linear 60) and the additional 4 switches in the box are more a tradition than a requirement. You will still be able to pull out the switches, of course, but we won’t provide a tool in the box to pull them with.

The plan is to either offer the switch puller when we have another switch variant available or when we’ve completed the development = we are satisfied. Whatever comes first. Kevin already started 3D printing different designs and kicked it off. When we have a final version, we will make the 3D print files available for anybody to print for personal use, and then make plans to open tooling.

We will have options available for existing keyboard owners to easily obtain the puller when it’s done. Talk about this another time.

The Timeline

At this stage, I’m fairly certain we won’t be slowed down by a part supplier anymore. We’re now increasingly dependent on the manufacturer’s schedule. There are only a few months left before Chinese New Year (CNY) on the 12th of February. This is when all factories closer for a fat month. Usually, the factory’s labor force shrinks the closer it gets to CNY and customers want to push out their last moment things. Now with COVID in play, it’s all a bit questionable.

I’m not entirely sure if I should be concerned by this in relation to the manufacturer’s schedule, but I have already noticed with the DVT that we have to wait our turn. The DVT could have, in theory, taken place in the first week of December. Everything should be available and ready by then. However, it’s scheduled for the 17th of December, because that’s the earliest available date to run all the test machines. In the meanwhile, we’re not allowed to schedule a mass production until after DVT is completed.

Is this me preparing you for the inevitable? No, not at all. The manufacturer is aligned to get it all done before CNY, I’ve hammered on this point, but it’s my inexperience with them and this scheduling that’s giving me pessimistic thoughts.

Right now there’s not much we can do anymore for the Wooting two Lekker Edition until DVT takes place. We’re all working as fast as possible, Kevin is being sharp AF to spot any issues, and I’m leaving nothing to chance. At the same time, we’re also getting the Wooting two HE ready for DVT in January.

Alright, the schedule:

Software

It’s great to see how Jeroen and Simon are making headways with the software. There’s a lot that they’re working on, but let’s break it down to these:

• Analog key calibration
• Adjustable actuation point
• Rapid trigger
• Dynamic key scanning
• RGB Animations

In the meanwhile, Erik is working on the new Wootility concept that will include our new approach to memory and what you can do in profiles. It’ll also aim to make everything more explanatory and coherent, so it’s even easier and faster to understand. More on this in perhaps the next update.

Analog key calibration

Jeroen and Simon concluded that it’s smarter to take a lean approach towards key calibration. This means they’ll launch the first version with a generalized concept that works, but isn’t perfect. Then based on the data we collect from the production and feedback from end-users, tweak and formulate a better algorithm to improve it.

Jeroen is adamant to provide a painless and the least possible intrusive method to calibrate keys. Ideally, this happens automatically in the background, but without enough information, this can be a huge waste of effort.

What makes the calibration process a bit tricky is that the key is not a fixed Z-axis that only moves in 2 directions. The sensor detects in proximity, not pin accurate position. This is tricky at the end of the press where you can swirl the key around and therefore the proximity of the magnet without retracting the key.

To über simplify the calibration. You should imagine that the Hall effect sensor inside the keyboard outputs an (arbitrary) value of 0 to 100 based on the magnet proximity. 0 being no detection, 1 detected, 100 maximum. When we focus on a single key, the sensor can already detect the magnet and it will output a value of 5. When you press this key to the end, it will output a value of 96. When we start calibrating, we will link the 5 value to the first 0.0mm (resting position) and the 96 value to the end 4.0mm. The values in between are equally divided over a linear line.

However, the end value will vary depending on where or how you press the key. Press it on the left top side, it might en on 91, on the right side 98…. press the switch with force into the keyboard the value maxes at 100. If we look back at the calibration example, where 4.0mm is set to value 96, when you press the key on the left top side it might only reach 91. Which will then equal a distance of 3.8mm, instead of your expected 4.0mm. If you translate this to a joystick, it means you won’t run at full speed, but just slightly below, while for all you know, you’re pressing the key to the end.

There are 108 keys that each have their own start and end value.

The easiest solution to this is to add a nice deadzone and say that any value surpassing 90 is equal to 4.0mm, though it won’t be accurate for every key, it’ll be close enough and it’ll ensure functionality. An improvement on this is to take the end value from the calibration step minus 5, which better adapts to multiple keys and is more accurate. Calibration is a bit more complicated than these examples, as there are more factors to take into consideration. We hope by collecting more data on the hall sensor, switch variances, and user feedback we will gain more insight that might help us create smarter solutions. Ultimately, we’d want the keyboard to dynamically calibrate the switch on the fly based. If we can accurately determine your typing behavior, we can also tailor the response. But that’s future talk, not to mention that we can’t even collect that type of data at this moment and without consent.

Back to the start, we will provide options for novices to tinker around with the full range analog sensing without restrictions. But we will limit the options on the first launch to ensure a smooth experience for first-time users.

Adjustable Actuation Point

If you were a keen observer, you might have noticed from our Wooting two HE marketing material that we mention it has 0.1mm to 4.0mm adjustable actuation point. This is not bullshit.

https://youtu.be/BUC0S2OST1k

Just not really practical if you take either one of the extremes, haha. We decided to limit the range you can adjust unless you unlock the full range. This is, as mentioned before, to ensure first-time users have a smooth experience and won’t think it’s a defect. The standard limits will probably be 0.3 to 3.8mm.

Rapid Trigger

Simon isn’t completely done with Rapid trigger yet, there are constant tweaks required based on the other changes we made. In specific the keyboard scanning (last part), the faster we scan all the keys, the more responsive we can make rapid trigger. There’s a small Rapid Trigger demo that Jeroen made to clearly show how it works using the RGB LED. An idea from CrossVS on Discord.

https://youtu.be/Pz8F1h3bgXc

When it’s green the key is activated, when it’s red the key is deactivated. This basically means you can reactivate (trigger) the key anywhere in the key range rapidly! There’s a small mind-bender you need to understand though, what’s the impact of your adjustable actuation point on the Rapid trigger range?

Rapid trigger’s first activation starts at your configured actuation point, after which the entire switch range is utilized for Rapid trigger to reactivate. For example, if your actuation point is 2.0mm, your key’s first actuation is at the 2.0mm, then you can return the key to, for example, 0.5mm (it deactivates retracting for ~0.2mm), and then press it down again to activate it at the 0.7mm. You can repeat this exercise throughout the entire switch range until you entirely let go of the key. The first actuation starts at 2.0mm again.

We need to take a look to what extent you can tweak Rapid trigger without overcomplicating it and keeping it user friendly. There’s one obvious setting that you might want to tweak, the sensitivity. How far you should press or retract to activate or deactivate a key.

Parallel key scanning

This is a bonus part. The Lekker switches challenged Jeroen and Simon to innovate more on input speed. We want to make sure we can meet our Tachyon mode requirements and more. However, Hall-effect switches aren’t as straight forward as optical switches. In the last update, I covered how we had to deal with noise to find an ideal balance between signal resolution and input speed. The lower the resolution (meaning less gradual steps per mm), the faster the input speed, the higher the resolution, the slower the input speed. We strive to optimize both, not one or the other. This was a hardware decision that we can’t modify.

What we can modify however is software and how we handle all the hardware parts. Since the Lekker switches were more challenging to meet our <1ms tachyon requirement, Jeroen and Simon also tackled signal processing. It’s one of Jeroen’s ideas that I mentioned in the previous update.

In short, instead of scanning and processing each individual key input signal, we now scan and process multiple keys in parallel. The entire process now also runs in the background and can’t be interrupted by other processes (RGB), it has prioritization. These two improvements result in input speed gains. You won’t find noticeable gains while running Tachyon mode (though it is faster!), but you will outside of tachyon mode. It’s a software update that affects all our keyboards, including Flaretech keyboards.

Voltage versus bits

The following is a super simplification of what happens, steering clear from nitty details to keep it relatively short and comprehendible.

You need to familiarize yourself with the following glossary:

• Microcontroller (MCU)
• The brains of the keyboard. This is where all the calculations take place.
• Analog to Digital Converter (ADC) port
• A port on the MCU that allows analog (a voltage) signal input for the ADC converter.
• Analog to Digital conversion
• The process of converting an analog signal (a voltage) to a digital signal (bits).
• Multiplexer (MUX)
• Similar to a power switch, this expands the number of ports on a single power outlet. In this case, it expands the number of keys that can connect to a single ADC port.

When you press a key, there’s a voltage (analog signal) that’s outputted and needs to be converted into bits and bytes (digital signal) before it can be processed by any software on either the MCU or PC. This is why there’s an Analog to Digital conversion section on the MCU that solely converts analog signals to digital signals. This conversion process has 5 separate steps or calculations before it’s done. It can only process 1 analog signal at a time and it can only receive the analog input from ADC ports.

Wooting keyboard’s MCU has 8 ADC ports that can be connected to different voltage sources for the ADC conversion. Each key on the keyboard outputs a variable voltage and needs to be connected to an ADC port, otherwise, the analog signal can’t be converted into anything useful. It introduces a problem when a keyboard has 108 (Ansi layout) keys but the MCU only has 8 ADC ports.

Circuit designs

The majority of keyboards use a matrix design to connect all the keys over a limited amount of ADC ports. A matrix design organizes all the keys over a grid divided into rows and columns, just like a spreadsheet. You can create cross-references in the circuitry by actively reading a column and controlling the row you activate. A matrix design can be set up in a heap of different ways, with different consequences, but there are two basic issues. You can’t read multiple rows at the same time, because you can only receive a singular voltage value per port (column), and all the circuits are somehow connected to each other, which can create some funky or unpredictable voltage scenarios. The advantage of a matrix design is cost-efficiency. It requires fewer components and fewer ADC ports on an MCU. More ADC ports = more $$.

We use a less common method. Honestly, I have no idea who else does it like us. We use multiplexers (MUX) to expand 1 ADC port into 16 ADC ports. It creates a direct circuit with each individual key by quickly cycling through the 16 channels on the MUX. This means we can make a direct circuit connection for up to 16 keys per ADC port. We use 7 MUX, that’s good for 112 keys.

Alright, now you understand how everything is connected. Let’s focus on the reading process.

Polling and scanning

A keyboard polls for input. It actively scans each key for input, as opposed to the key sending the input immediately to the MCU. This is the same concept as USB polling. If you’re familiar with USB 1000Hz polling, then you know that USB will check each connected device every 1ms for a new data packet. For reference, 500Hz is 2ms, and 125Hz is 8ms. At 1000Hz your input is refreshed every 1ms with whatever data is presented at the moment it polls.

On a keyboard, this concept is exactly the same but instead of devices, it’s each key. Whatever the status of the key is at the very moment it polls, is also the status that is sent to the MCU for processing. Regardless of what happened before the poll or after the poll, until the next poll. Any status in between polls is lost. This is why a high polling rate on your keyboard will result in much smoother analog input movement and is more desirable for the fastest input. We refer to the keyboard’s polling rate as the keyboard’s scanning rate to avoid confusion with USB polling.

To summarize everything, you now know that:

• Keys output a voltage (analog signal) that needs to be converted to bits and bytes (digital signal) before it can be used for anything.
• A Microcontroller (MCU) has Analog to Digital Converter (ADC) ports that accept voltage input from the keys.
• The MCU can only process 1 analog to digital conversion at the same time.
• The analog to digital conversation takes 5 steps to complete.
• There are often far fewer ADC ports than keys, which requires a Matrix, Multiplexer (MUX), or other design to connect multiple keys to a single ADC.
• The MUX expands the ADC ports and still retains a direct circuit connection with each key.
• The MCU actively scans keys for input and can only read the exact status of the moment it scans.

Now that you understand the basic mechanisms, let’s put everything together.

The old design

In our original design we would activate 1 MUX, scan a key, if status was “inactive” move on to the next key, if status was “active” (is pressed), process it in the ADC converter, wait until all 5 steps completed, move on to the next key, repeat cycle until all keys on MUX were scanned, move on to next MUX, repeat the cycle until all keys were scanned, and start all over again.

When you turned on Tachyon mode, the scanning rate went faster, and regardless of how many keys you’d press, it would complete a full cycle within <1ms. Optical switches are very effective, they need very little time to activate before you can read a voltage.

Hall effect sensors however take a little bit longer to activate before you can read a voltage and it created a bottleneck in the scanning cycle. It helped us emphasized that our scanning design should be done more efficiently.

The new design

In our new design, we activate 4 MUX simultaneous, scan 1 key per activated MUX simultaneous, if the status is “inactive” ignore it, if the status is “active” send it to the ADC converter step 1/5, ADC converter processes it in step 1 and moves it to step 2, continue scan, next active key is sent to ADC converter step 1/5, ADC converter processes it in step 1 while at the same time it moves step 2’s signal to step 3, complete process in step 1 and move it to step 2, continue scan, repeat cycle until all keys on 4 MUX are scanned, move to final 3 MUX, repeat cycle until all keys were scanned, and start all over again.

There are 2 major improvements in this new design:

1. Multiple keys are scanned in parallel, therefore a scan cycle can complete much faster.
2. The ADC converter now processes multiple analog signals by dividing them over the 5 steps. This shortens the processing time per signal. It’s similar to a product assembly line, each step towards the end completes another part of the final product.

Lastly, there’s one more improvement. This process always runs in the background and has priority over everything else. This means you’ll get consistent fast input, regardless of what’s happening on the keyboard. This is important when you run for example RGB effects, very intense effects could fight for computing power at very high scan rates and clog processes. This is why RGB effects are disabled in tachyon mode. Now that scanning and input processing has priority, RGB effects shouldn’t have a huge impact anymore, but, for now, we’re still leaving it disabled in Tachyon mode.

This new design does not solve the phenomena described in the previous blog. The more keys you press simultaneously, the longer it takes for a full scan cycle to complete. It does improve the total result and decreases the increment per key pressed. I can’t publish any accurate numbers yet, but in general, we aim for Tachyon mode, 10 keys simultaneous, input <1ms.

Lekker Edition Lekker Edition Lekker Edition

Just to remind you, we’re still talking about the Wooting two Lekker Edition and this update is much longer than I anticipated.

Anything that stands in the way of manufacturing and putting those goods on a boat out of China before Chinese New Year will be murdered by me. I’m confident with the technology, quality, and final product. It’s now really time to get it in your hands. We need to move forwards!

If you read this without either a Wooting Lekker Edition or HE, then all I can do is recommend you get a Wooting two HE now. I’m afraid we’ll quickly run out of stock by the time we can deliver the Wooting two HE, then there’ll be a big gap before we can restock. As for how things stand now, we need to plan these productions 4-6 months ahead of time. Silicon manufacturing demand is skyrocketing in Taiwan and clogged in 2021. There are a few components’ lead time that runs into 188 days (6 months) in 2021. We try to stock components with long lead times in advance, but there’s a limit to what we can finance in advance.

Expect another update near the end of December, after DVT on the 17th of December. I’m looking forward to it.

Much <3,

Calder