Tuesday, September 15, 2015


I've gone ahead and started butchering the light cables.
And I only killed seven switches in the process...

Man, it kinda hurts...but it will be all worth it when finished! :)
I will be able to say "light #5, that's red/blue" and see if the problem is the bulb, the board or MOSFET's etc. Troubleshooting will be a lot easier!

1) Old cables removed and the new ones await installing!

2) They don't look much, but will mean the world to Mr Bubbles!
I might need to order another batch of cables thou - apparently 60 m isn't enough ...

Sunday, September 13, 2015

Hmmm-stable? Unstable?

So, long story short -
I've struggled with random crashes and potential memory corruption scenarios for a while. But now I believe I've nailed down all bugs, starting from the String library and ended with the sound card serial commands... 

1) This is what most of my evenings look like nowadays. Can't say I'm not enjoying programming the machine, but a proper debug mode with breakpoints would be nice...

What I have done lately:

+ Rewritten most of the core
+ Written tons of rules
+ Changed switches, solenoid and rendering to interrupt driven updates on Timer3 (tried Timer5 as well, no difference)
+ Added a large const array of animation information. (tried to cut it in half, made no difference for the crashes)
+ Added communication over Serial1@57600 to wavTrigger and Serial3@250000 to Arduino Mega2560
+ Added softPWMServo instead of hardware servos
+ Added EEPROM reading/writing of highscores and settings
+ Switched to dynamically allocated data (this was AFTER the crashes were observed, and it's never deleted/recreated - only once during startup)

I've switched all String's to const char* and...ta-ta - const Strings (sorry Majenko!) as well as some generic safety precautions. I've measured RAM during idle and play and it does indeed stay fixed now. If I do a "new" without delete the available RAM ticks down and once low enough it crashes, so the RAM function seem to work. There were just too many functions and special occasion that I needed to replace, and by doing so I'd most certainly would have ended up with an even buggier version of the machine. So once I saw that RAM usage was static, I moved on.

I've also implemented the hardware watchdog so in case it does crash it will reboot and it won't start burning (tried that, wasn't pretty). Works pretty great too! A peculiar thing thou - if the watchdog was held alive by an interrupt, the main loop could crash and the interrupt continue. Don't ask me how, but it did. I did a divide-by-zero on purpose and all game loop and switches etc froze, but the rendering kept going. So the watchdog seems like it must reside in the main loop to have a proper effect.

Lastly, what still caused crashes after everything else was fixed was actually the sound card, a Wav Trigger.

If I sent "too many", i.e more than a few request per 10-15 millisec it would eventually crash. I'm not sure which end causes the problems, but I do know that I can crash the machine by sending a faulty command to the sound card. (RobertSonics, if you're reading this - there could be something wrong with the latest firmware. It shouldn't crash if the command is formatted correctly but unknown...But then again - Chipkit shouldn't crash if the sound card crashes either...)

What I've done is to simply add a cooldown on each command sent to the sound card so that it will sit and wait for it's turn and then send the command. This works and no sound is every skipped/missed, but the implementation is not nice as everything except for solenoids and rendering is stalled in case there's lots of audio. Realistically there's probably only ever gonna be 3-6 samples being started every 300 ms during play, so perhaps I can add the cooldown only after a certain amount of active tracks instead of every time, for instance.

On top of this I've added animation, sound and rules to most standard switches now. I'll try to get this down in a video soon as it's looking pretty good, if I may say so myself. :)


Well, hello there, Mr Big-pile-of-cables...
Should we get you all soldered up?

Wednesday, September 2, 2015

Time flies!

Apparently I'm using a really really old version of MPIDE to compile the code with....
The servo-problem was solved a while back:

  12/21/2013 <BrianSchmalz>:
    * Fixed bug that caused glitches every 107 seconds.

The question is - do I dare to update the core software, considering the numerous changes I've made to the stock libraries? Can't say I recall exactly what I've changed and where.

But I suppose I need to do this someday anywho since the laptop I'm using is a very old one and can't even compile the code with optimizations active. For the release version I'm pretty sure I want to use compiler optimized code. :)

Also, a programmer called Majenko suggested that the Arduino cannot reliably communicate over serial at higher speed than 9600 baud. I'm not sure what to believe here, since I've used much higher speeds - but still, Majenko is a great programmer and has supplied a lot of libraries and corrections, so I figure he/she knows better. If the serial interface used to program the device is different than the actual interface the MCU is using for the pins inside sketches this might explain why the sketch is running great connected to my computer while failing connected to the MCU.

"The Arduino is unable to communicate reliably at higher baud rates due to the clock being unable to accurately divide down to the right kind of values, but the chipKIT boards, because of the higher speed master clock, can do it more accurately. As a result the Arduino sometimes can't communicate through serial with the chipKIT boards. Try running at a lower baud rate (9600)." [Chipkit forum]

9600 baud could possibly be enough for my needs, but I'm not sure what the "damage" to SD-reading and frame building would be. Still, it's something that can be investigated in the future.


I've installed the sound board and light board now (light cables are still not done thou).

There's a slight ghosting issue present, hopefully it won't be disturbing with the proper lights as I've seen commercial machines do this as well, if you look closely enough. The problem is caused by the weirdest error ever - the lights are running perfectly when connected to a computer and receiving serial input. But when connected to the Chipkit it just goes bonanza. Naturally I suspected the Chipkit first due to recent interrupt changes, but the sound board worked fine on different serial ports and speeds so there's no problem with the serial ports, of that I'm sure.

I realized I was using a rather long (with delayMicroseconds()) interrupt routine on the Arduino and this caused problems with the serial input - but only when connected to another microcontroller... I shorted it down and now it works with the Chipkit, but is instead plagued with ghosting. Which it really shouldn't be doing. It's pretty weird, but for now this "workaround" will do great, but I'll  revisit this in the future if it turns out to be a proper problem. 

I've also programmed highscore saving/loading as well as proper settings saving/loading and a hardware alert routine during boot. The servos started acting up as a result of the interrupt driven display so I replaced the interrupt servo routine with a software based one instead. Seems to work well, except for an occasional servo "tick" now and then on the first out of two servos. This can probably be avoided by doing some magic in the library-code, as I suspect it's dropping due to a timer reset or similar. Quite possibly a dummy servo can be configured to take the hit instead...

Overall the machine is starting to feel really slick now and the software is bumped up to version 0.85!

Monday, August 31, 2015

Major Code and Sgt Refactor

I haven't gotten the motivation yet (damn weather!) for soldering the new light cables, but I have spent some time with the code the last days. Figured I could do something better than what was in place.

Said and done -
Previously the code was run sequentially and always in the same order. This has now been changed to be done via interrupts instead. I am drawing line by line, layer by layer, and it works great!

I've also sacrificed a little RAM for speed/comfort by adding a backbuffer. While the display is blazing away as fast as it can from the drawbuffer (i.e frontbuffer), all new drawings occur on the backbuffer. When the buffer is finished it simply redirects the current line-pointer to the other buffer and the display never knew what happened. I've tried this in the past, but I never got the flickering to go away. I realized I was going about it the wrong way - there's no need to restart the drawing from the top whenever the content changes. I'm already drawing the screen line by line times eight so nobody would notice really. I was also previously stalling the rendering for buffer creation due to the sequential handling and that caused flickering too. But by avoiding the above and rendering with double buffers, interrupts has given me a rock solid 60 fps screen that is flicker free and brighter than it's ever been. Best of all - it's fixed fps, so it will never dip below this. Ever.  

I've also moved solenoid and switch handling into the very same line-drawing-routine, except that I only process information from these once 32 lines has been drawn (i.e at vertical blank, like the good old days). This has almost given me a 100-200x frequency on I/O without causing any flickering!

This is extremely good news as this means less chance of a missed switch hit or a solenoid firing too long. Previously, if the rendering and SD-card loading took 30ms, that was 30ms that couldn't be spent on anything - i.e a switch hit would not be registered during that time. That is remedied now thou and just to be extra sure I rewrote the switch code to be 4-5x faster as well. Turned out that piece of code still used the old and slow digitalWrite-code, but I took care of that. 

Furthermore - with the changes above the logic loop didn't have to be running as fast as possible. So I've locked the game logic to 40Hz, including SD-card reading, buffer creation, transitions etc. I did a little counting and found that it never dipped below 42 Hz (this is all done on the machine's "spare time" now, mind you), so I lowered it a little extra for safety. A fixed rate means no surprises to game logic and the frame is guaranteed to be finished in time. Should it not be, however, the double buffering will prevent any flickering - which is always nice!

For the record.
I've tried using interruptTransfers in the DSPI-library, but they refuse to work for me due to trouble with the Max32 board definition and conflicting hardware vectors. Ironically, I'm the one who reported the bug in 2013 and it hasn't (to my understanding) been fixed yet...

Anyhow - had that been fixed I could have load animation data and sending data to the display "in the background" and go about my business the usual way. My way works just the same, but different.

Finally, I wrote a script to remove the initial two bytes of junk from wav-files. However, as I didn't read up on the wav-format, the files ended up being the correct size but not valid wav-files. Instead of working around it I simply powered through and converted the 636 sound files (will they all be used anyway?!) manually. That was joyful, for sure, but now the audio seems to be working fine. Still need to convert the old audio routines into new ones and decide on a sound priority/scheduling scheme.

Baby steps...

Thursday, August 20, 2015


I found this pretty interesting -
This is a (blurry) shot of the inside of a Briarwood Aspen pinball machine. That's a commercial game from late 70's.

1) Briarwood Aspen inside. Not much in there to be honest...
Compared to my own pinball that machine looks like the homebrewed one... 

2) My own BioShock pinball. The cabling is arguably neater in the BA machine thou.
But in my defense, so was mine once... :)

On the other hand - had I been making an late 70's/early 80's machine I'd be done by now....

Two Bytes

Two bytes were enough to break the WAV-Triggers loading...

When I did the export of the WAV-files I pressed the "Clear metadata"-button in Audacity. It seems this buttonpress added a "no metadata"-flag, or something like that and thus causing the file to not function in the WAV Trigger.

When exporting the file again and simply pressing "Export" the file is 2 bytes smaller and works.

I'm currently toying with the idea of either exporting all files manually or doing a crude batch script to remove the first two bytes of all sound files...