Roomba 980 not booting

Inside the Roomba and Scooba and more, Cool mods, Repair and Upgrades - including the all new iRobot Create Kit. Let's void that warranty baby!
Post Reply
StefanKrupop
Posts: 5
Joined: June 14th, 2019, 8:58 pm

Roomba 980 not booting

Post by StefanKrupop »

Hi all,

I bought a defective Roomba 980 off eBay and am now trying to fix it. From the outside, there is no sign of live in the unit, so no beeps or LEDs or anything, even if I directly provide power from a lab supply to the battery contacts.
However, the bootloader in the STM32 (U6) seems to be at least partially working, as I get the following UART output on J25:

Code: Select all

External flash manufacturer ID is 0x9d6017
External flash unique ID is 0x5031473635370009

Normal reset

iRobot Roomba Bootloader svn version: 4042
USB PID: 2
option bytes read protect:  1
option bytes write protect: 0xfffffc00
option bytes user:          0xff
usb_activity_timeout_ms: 30000
No connection in 30 seconds, booting the app
This repeats over and over again, it never actually "boots the app" but rather restarts. I desoldered both flash chips (U11 for the STM32 and U12 for the Wifi module) and dumped them externally. Content seems to be relatively ok (some interesting strings).

I then attached a logic analyzer to U11 and recorded what the bootloader actually does with the flash. It reads the JEDEC ID (shown as the "manufacturer ID" in the bootloader output), the unique ID and then the first 511 bytes of the flash. After that it seems to reset.

So I conclude that something might be wrong with my first bytes. One thing I already see is that my external programmer and the SPI log disagree on the 3rd byte: The programmer reads it as 0x00 (verified multiple times), while the trace shows 0xff is returned.

I'll try to fix that, but I was wondering whether somebody could help me out with a dump of their first 512 bytes of U11? For me it's all 0xff except the first 8 bytes. There I have

Code: Select all

0x01 0xff 0x00 0xff 0x00 0x00 0x01 0x00
. Bytes 5-8 could be the start address of the firmware image in the flash. But I do not know what the bootloader expects to be in Bytes 1 to 4...

Thank you very much!

Kind regards,
Stefan Krupop
Last edited by StefanKrupop on March 23rd, 2020, 6:56 am, edited 1 time in total.
ps0001
Robot Addict
Posts: 283
Joined: December 5th, 2017, 3:25 pm

Re: Roomba 980 not booting

Post by ps0001 »

I don't have a 980 I can use to help you anymore, but I wanted to say I'm impressed you made it this far! Would you mind sharing the interesting strings you found in the flash chips?
StefanKrupop
Posts: 5
Joined: June 14th, 2019, 8:58 pm

Re: Roomba 980 not booting

Post by StefanKrupop »

ps0001 wrote:Would you mind sharing the interesting strings you found in the flash chips?
There's lots and lots of internal function names and debug output, literally thousands.
Here are some of the more interesting:

Code: Select all

firmware version: 01.12.01#1
Camera switching to SLAM mode.
Camera switching to streaming mode.
Camera switching to QR-code mode.
Roomba 9xx Camera Test
cam-test: Found center QR code, success!
cam-test: %s: area too small: %f < %f
cam-test: got top left
cam-test: got top right
Bottom Left
cam-test: got bottom left
Bottom Right
cam-test: got bottom right
cam-test: Got all corners, success
void blahblahblah::DockTracker::addObsvHelper(unsigned int, unsigned int, bool, const Vector2*, const Vector2*)
ersp_uint32 blahblahblah::DockTracker::centerDockField()
Strategy: starting evade, shine on you crazy diamond
Strategy: evade claims success, a great day for freedom, %d
Strategy: evade failed, re-try, %d
Strategy: reached max evade failures, comfortably numb
Strategy: blocked-off, check do-able actions
Strategy: no reachable snakes, switch to pwf
Strategy: plan to go to unreachable snake
Strategy: no reachable pwfs, switch to open area
Strategy: no reachable pwf/snake, prefer going to snake
Strategy: plan to go to unreachable pwf
Strategy: inc force plan count %d
Strategy: reached max force plan, abort.
Mouse: in the beginning there was a sensor...
Mouse: sync with prod_id
Mouse: Bad product id: %d
Mouse: got correct product id: %d
Mouse: reset sensor
Evolution Robotics
LPC3250 Board
imager_ov7740: could not open device %s
Aspen firmware ver %d '%s'
A28ME8NCAI1WG2.iot.us-east-1.amazonaws.com
Received Kiss o' death from server
[%d] MQTT: FSM-%p[%d] need=%d got=%d rem=%d
/mapUpload
/mission/report
/deployment
/shadow/update/delta
/shadow/update
https://disc-%s.iot.irobotapi.com/v1/robot/discover?robot_id=%s&country_code=%s&sku=%s
Successfully sent the map to AWS S3
lwip_connect: invalid address
-----BEGIN CERTIFICATE-----
MIIE0zCCA7ugAwIBAgIQGNrRniZ96LtKIVjNzGs7SjANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp
U2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxW
ZXJpU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0
aG9yaXR5IC0gRzUwHhcNMDYxMTA4MDAwMDAwWhcNMzYwNzE2MjM1OTU5WjCByjEL
MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW
ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJpU2ln
biwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJp
U2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9y
aXR5IC0gRzUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvJAgIKXo1
nmAMqudLO07cfLw8RRy7K+D+KQL5VwijZIUVJ/XxrcgxiV0i6CqqpkKzj/i5Vbex
t0uz/o9+B1fs70PbZmIVYc9gDaTY3vjgw2IIPVQT60nKWVSFJuUrjxuf6/WhkcIz
SdhDY2pSS9KP6HBRTdGJaXvHcPaz3BJ023tdS1bTlr8Vd6Gw9KIl8q8ckmcY5fQG
BO+QueQA5N06tRn/Arr0PO7gi+s3i+z016zy9vA9r911kTMZHRxAy3QkGSGT2RT+
rCpSx4/VBEnkjWNHiDxpg8v+R70rfk/Fla4OndTRQ8Bnc+MUCH7lP59zuDMKz10/
NIeWiu5T6CUVAgMBAAGjgbIwga8wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E
BAMCAQYwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJaW1hZ2UvZ2lmMCEwHzAH
BgUrDgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4wJRYjaHR0cDovL2xvZ28udmVy
aXNpZ24uY29tL3ZzbG9nby5naWYwHQYDVR0OBBYEFH/TZafC3ey78DAJ80M5+gKv
MzEzMA0GCSqGSIb3DQEBBQUAA4IBAQCTJEowX2LP2BqYLz3q3JktvXf2pXkiOOzE
p6B4Eq1iDkVwZMXnl2YtmAl+X6/WzChl8gGqCBpH3vn5fJJaCGkgDdk+bW48DW7Y
5gaRQBi5+MHt39tBquCWIMnNZBU4gcmU7qKEKQsTb47bDN0lAtukixlE0kF6BWlK
WE9gyn6CagsCqiUXObXbf+eEZSqVir2G3l6BFoMtEMze/aiCKm0oHw0LxOXnGiYZ
4fQRbxC1lfznQgUy286dUV4otp6F01vvpX1FQHKOtw5rDgb7MzVIcbidJ4vEZV8N
hnacRHr2lVz2XTIIM6RUthg/aFzyQkqFOFSDX9HoLPKsEdao7WNq
-----END CERTIFICATE-----
A28ME8NCAI1WG2.iot.us-east-1.amazonaws.com
0.irobot.pool.ntp.org 1.irobot.pool.ntp.org 2.irobot.pool.ntp.org 3.irobot.pool.ntp.org
-----BEGIN CERTIFICATE-----
MIIEDzCCAvegAwIBAgIBADANBgkqhkiG9w0BAQUFADBoMQswCQYDVQQGEwJVUzEl
MCMGA1UEChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMp
U3RhcmZpZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDQw
NjI5MTczOTE2WhcNMzQwNjI5MTczOTE2WjBoMQswCQYDVQQGEwJVUzElMCMGA1UE
ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMpU3RhcmZp
ZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggEgMA0GCSqGSIb3
DQEBAQUAA4IBDQAwggEIAoIBAQC3Msj+6XGmBIWtDBFk385N78gDGIc/oav7PKaf
8MOh2tTYbitTkPskpD6E8J7oX+zlJ0T1KKY/e97gKvDIr1MvnsoFAZMej2YcOadN
+lq2cwQlZut3f+dZxkqZJRRU6ybH838Z1TBwj6+wRir/resp7defqgSHo9T5iaU0
X9tDkYI22WY8sbi5gv2cOj4QyDvvBmVmepsZGD3/cVE8MC5fvj13c7JdBmzDI1aa
K4UmkhynArPkPw2vCHmCuDY96pzTNbO8acr1zJ3o/WSNF4Azbl5KXZnJHoe0nRrA
1W4TNSNe35tfPe/W93bC6j67eA0cQmdrBNj41tpvi/JEoAGrAgEDo4HFMIHCMB0G
A1UdDgQWBBS/X7fRzt0fhvRbVazc1xDCDqmI5zCBkgYDVR0jBIGKMIGHgBS/X7fR
zt0fhvRbVazc1xDCDqmI56FspGowaDELMAkGA1UEBhMCVVMxJTAjBgNVBAoTHFN0
YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xMjAwBgNVBAsTKVN0YXJmaWVsZCBD
bGFzcyAyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggEAMAwGA1UdEwQFMAMBAf8w
DQYJKoZIhvcNAQEFBQADggEBAAWdP4id0ckaVaGsafPzWdqbAYcaT1epoXkJKtv3
L7IezMdeatiDh6GX70k1PncGQVhiv45YuApnP+yz3SFmH8lU+nLMPUxA2IGvd56D
eruix/U0F47ZEUD0/CwqTRV/p2JdLiXTAAsgGh1o+Re49L2L7ShZ3U0WixeDyLJl
xy16paq8U4Zt3VekyvggQQto8PT7dL5WXXp59fkdheMtlb71cZBDzI0fmgAKhynp
VSJYACPq4xJDKVtHCN2MQWplBqjlIapBtJUhlbl90TSrE9atvNziPTnNvT51cKEY
WQPJIrSPnNVeKtelttQKbfi3QBFGmh95DmK/D5fs4C8fF5Q=
-----END CERTIFICATE-----
The "Received Kiss o' death from server" worried me at first, but I think that's just from the NTP client and not really killing the robot :wink:

Stefan
ps0001
Robot Addict
Posts: 283
Joined: December 5th, 2017, 3:25 pm

Re: Roomba 980 not booting

Post by ps0001 »

Neat, thanks!
StefanKrupop
Posts: 5
Joined: June 14th, 2019, 8:58 pm

Re: Roomba 980 not booting

Post by StefanKrupop »

FINALLY!!! :dance:
After almost a year trying now and then I finally was able to make quite some progress on the Roomba.
The following, lengthy, paragraph will be a description of how I managed to come to the current state. Feel free to skip it if you're not interested in the technical details of dumping the firmware.

Some time after the last post I got a U11 from a Russian guy. It was (basically) identical to mine, so it was not U11 preventing it from booting.
So it was clear that I needed the bootloader code to understand why it was resetting. Unfortunately, the main STM32F1 microcontroller of the roomba is, of course, read-protected. I did not even dare to ask iRobot whether they could help me out with a binary image of it. Thus started the journey to circumvent the read-out protection of the STM32F103ZET6 (U6). After trying some software-only attacks unsuccessfully, I found the paper "Shaping the Glitch: Optimizing Voltage Fault Injection Attacks" that indicated a power glitching attach against the STM32F1 series might be feasible. I tried to implement it using different techniques over the next month, but my CPU-based attempts all failed, even using the signal generator mentioned in the paper.
But finally, last week I had the big breakthrough: I finally decided to dust off my rusty VHDL skills and implement the necessary bootloader communication and glitching logic inside a FPGA, using my old Altera DE1 board. I used a simple hardware approach for the glitch circuit: Just feeding power to the (removed from the Roomba) STM via a 15 Ohm resistor and shorting that to ground when glitching using a IRLR2905 FET. Boy was I happy after the bootloader replied 0x79 instead of 0x1F for the first time, after fiddling around with different glitch positions and lengths using the switches on the DE1 for a while! This showed the glitch was actually working. After implementing the rest of the read command in the FPGA it actually returned the first 256 bytes of the read-protected memory! :mrgreen: A little while later I had extracted the full 512 kB of the firmware and could finally start debugging my boot problem.

To do so I decided not to reflash the orignal STM (to remove the debugging lock), but instead use a new chip, so I would be able to re-read the original one, if necessary. What should I say? After soldering in the new processor and flashing it with the extracted firmware the Roomba just came to life! :dance: No further debugging required, it just booted, charged and would start a cleaning cycle.

Of course I also wanted to use the App to set the robot up. Sadly, this did only work partially. It would see the Robot and try to provision it, but then cancel saying that there's a "bad certificate".
This then led me to the discovery that the firmware actually uses the "Unique device ID" of the STM32 microcontroller to generate the "Bootloader ID". This BLID is also included in the SSL certificates stored in the robot and presented to the app. As I now have a different STM chip inside the Roomba, of course the Unique ID has changed (and is not changeable), which in turn changed the bootloader ID. Because it no longer matches what the certificates say, connecting fails.
So this is where I am currently working on. I could patch the firmware to return the Unique ID of the original controller (which I also read from it), but the firmware image is checksummed on boot, so I would need to understand the algorithm and fix the checksum. Also, this would break with the next firmware update, as it seems the calculation is actually done in the main firmware part and not the bootloader part itself. The other way would be to exchange the certificates sent against new self-signed ones, as I currently think the App is not really checking against the same root cert, but just checks the issuer string and that it is valid against the root cert presented by the robot. I have not yet checked whether the firmware containing the certificates is also checked for changes.

Kind regards,
Stefan
Attachments
Detail of the glitch
Detail of the glitch
Glitch after sending read command
Glitch after sending read command
My glitching setup. STM soldered to breakout board, just bare minium pins connected (VCC (+analog), GND, Reset and UART).
My glitching setup. STM soldered to breakout board, just bare minium pins connected (VCC (+analog), GND, Reset and UART).
STM32F103 removed, showing where traces underneath go ;-)
STM32F103 removed, showing where traces underneath go ;-)
User avatar
vic7767
Robot Master
Posts: 15556
Joined: January 14th, 2006, 7:31 pm
Location: Haughton Louisiana - USA

Re: Roomba 980 not booting

Post by vic7767 »

You've got some skills there. Let us know when you get a solution that allows for a way to exchange the certificates.
StefanKrupop
Posts: 5
Joined: June 14th, 2019, 8:58 pm

Re: Roomba 980 not booting

Post by StefanKrupop »

Thanks.
Current status: I checked the App, it actually checks the certificates against the root certificate it contains (the same root as in the robot) - as it should. So I think changing the certificates is not really viable anymore, as I also would need to replace it in the App (which would be possible).
So far I just patched the App to ignore the certificate not matching the BLID, which made it work. I was able to set up the robot.
I think I will still try to change the firmware to return the original unique ID instead of the one read from the uC. As far as I have seen I already have the newest firmware (v2.4.6-3) and there was no newer one for two years. So the risk of having to patch it over and over again seems low. So now I have to understand the Checksum mechanism first.

Kind regards,
Stefan
StefanKrupop
Posts: 5
Joined: June 14th, 2019, 8:58 pm

Re: Roomba 980 not booting

Post by StefanKrupop »

Ok, fixing it in firmware was not too bad.
The checksum algorithm is the same as described here by gssincla. Starting from Memory Address 0x0800A000 (so 0xA000 into the flash) it checksums all the way to the end of the flash. The checksum has to be 0x0 for the Bootloader to proceed. This is achieved by changing the last two bytes in the flash to the checksum result returned when ignoring those.
With that in place I was able to modify the FW. I kept it simple and just changed the constant pointing to the Unique ID memory location to instead point to almost the end of the flash where no more data was stored. There I just put the bytes I had read from the old STM.
After updating the CRC and flashing it to the robot now the BLID matches what is in the certificates and I could connect with the stock App directly from the Store without any issue. 8)
I checked some of the built in tests and now the Roomba is finally doing it's first cleaning cycle. So far it's looking normal, I would say.
That's it for me, defective Roomba finally fixed after almost a year. Money (even defective wasn't too cheap) not wasted. :mrgreen:

Kind regards,
Stefan
User avatar
vic7767
Robot Master
Posts: 15556
Joined: January 14th, 2006, 7:31 pm
Location: Haughton Louisiana - USA

Re: Roomba 980 not booting

Post by vic7767 »

Thank you for the update and information. :cheers:
ps0001
Robot Addict
Posts: 283
Joined: December 5th, 2017, 3:25 pm

Re: Roomba 980 not booting

Post by ps0001 »

Excellent work!
Post Reply