Tuning 2x PID Loops

Moderators: TomKerekes, dynomotion

Matt303
Posts: 8
Joined: Wed Jun 17, 2020 7:20 am

Tuning 2x PID Loops

Post by Matt303 » Wed Jun 17, 2020 7:43 am

I'm having issues tunning Kflop interfaced to an Odrive https://odriveroboticsdotcom The test set up is a nema17 8 pole BLDC motor connected to 1800ppr encoder. If I set it up with the SnapAmp I can get it running perfectly following the motion profile. I'm controlling the Odrive via PWM and direction pin. I have tuned the velocity control loop on the Odrive, when set at 0 velocity it's jumping one count on the encoder and really smooth up to 500000 c/s and responds instantly. I thought just having a P gain set in the Kflop would work, but it barely rotates when trying to follow the motion profile. I have bought the P gain up to the point it is unstable and dropped it back off. If I put in a small amount of I gain it rotates a lot faster but unstable on 0 position. Jumps around 10-20 count's and can't get it to steady with any I gain at all. If I back off the Odrive PIDs so they not so tight, I can increase I and P on Kflop a bit more, but it's unstable. It's like they are both fighting each other and been at it for days with no success. Here is a link to post in Odrive forum of issues [https://]discourse.odriveroboticsdotcom/t/motor-noise-and-vibrations-with-pids-at-0-gain/4926

User avatar
TomKerekes
Posts: 2529
Joined: Mon Dec 04, 2017 1:49 am

Re: Tuning 2x PID Loops

Post by TomKerekes » Wed Jun 17, 2020 1:56 pm

Hi Matt,

I doubt the problem is having 2 PID loops.

I suspect there may be a problem with using PWM to command Velocity. Does the ODrive have specs on the PWM input? What frequency? What mode? Sign and Magnitude? One problem with Sign and Magnitude is there can be distortion or deadband around zero. The circuity can have rise time/fall times, or other delays where small pulses are lost so the Drive "sees" zero up till a significant point then suddenly "sees" a value. You might try commanding constant values of 1, 2, 3 etc to see how the ODrive responds. Check if it is linear. Antiphase PWMs don't have this issue they output 50% for zero velocity.

The PWM might also introduce some time delay. Does the Drive average or filter it somehow. I wasn't able to find any specs.
Regards,

Tom Kerekes
Dynomotion, Inc.

Matt303
Posts: 8
Joined: Wed Jun 17, 2020 7:20 am

Re: Tuning 2x PID Loops

Post by Matt303 » Thu Jun 18, 2020 1:02 am

Hi Tom,

I started thinking about it also and I think you may be right. The PWM on the Odrive is a new implementation that is being worked on and I have merged it into my Github and devel branch of the Odrive. The details of the PWM can be found here further down the page https://github.com/MatthewReed303/ODriv ... _input.cpp I have tried at different PWM frequency, 1khz, and 2.5khz. Also here is a discussion on the actual PWM with the developers and gut who wrote the PWM code. https://github.com/madcowswe/ODrive/pull/292 and here are a few more details at the bottom of the doc page. https://github.com/MatthewReed303/ODriv ... erfaces.md I think the PWM response on the Odrive may just be to slow. Hardware-wise it's at 3.3v level pin from Kflop straight to pin of the STM32 microcontroller, no filtering or anything. I have confirmed clean signal from Kflop to Odrive with scope. I'm going to look into disabling some timers and system resources on the Odrive for the 2nd Axis and see if that helps PWM response time on the Odrive. Reading back through the PWM discussion I see mentioned
I think it would be better to set the timeout period to something like max_pwm_period * 120%. Otherwise, there is a 5ms delay when jumping from >0% duty cycle to 0% duty cycle. I would be surprised at this as a user, more surprised than having to configure a nominal frequency.
We can for instance run this check on tim_update_cb (low_level.cpp) which runs at 16kHz (8kHz after the first if). That way we can keep separate periods for each PWM input and don't have to use FreeRTOS timers.

REPLY
You're right, having the user set a nominal PWM frequency instead of trying to detect it would get rid of a lot of this logic.

The timeout for jumping from >0% duty cycle to 0% duty cycle is actually PWM_INACTIVE_TIME, set to 45ms, so it's worse than you thought. 45ms works for RC PWM, but for duty-cycle PWM it should be set relative to the nominal frequency as you said.
Would this 45ms time be causing any issues? I will have a look through the code and see if I can change it and recompile and test

Matt303
Posts: 8
Joined: Wed Jun 17, 2020 7:20 am

Re: Tuning 2x PID Loops

Post by Matt303 » Thu Jun 18, 2020 5:53 am

Hi Tom,

It looks like I have damaged the Kflop. The ground wire between the Kflop and Odrive disconnected and I wondered why the Encoder stopped reading all of a sudden so I grabbed a multimeter and checked between 3.3v and encoder line and all of a sudden there was a high-frequency noise and all lights went out and disconnected from PC and no longer powers up or anything. The Odrive board is fine and didn't do any damage which is strange it was only the Kflop that got damaged, if that was the reason or just happened to be something else that happened at the same time I was testing pins, the first encoder pin was fine, only happened when I moved the probe to the next pin. The Kflop was being powered from the USB port and had the jumper in place, and the USB port is still fine on the PC. I can't see any fuses on the board so it looks like I have fried it, is there anything I can test? Or is it history? I don't see how just measuring between encoder line and 3.3v could fry something or just bad luck and was actually something else.

I have tested the voltage regulators and the ld1086-33 is 3V and the L1117-25 is 2.6v and the L1117-12 is 2.35v which should be 1.2V could this be the issue? Have you got schematics so I can trace through and test components?

Thinking about it the 3.3v is the 3.3v supplied from the Odrive I was measuring. The Odrive has 3.3k pull up resistors to 3.3v and I have encoder going into Odrive and Kflop and just been sharing common ground between the boards and this became disconnected. I still don't see how this would have fried the Kflop as the encoder pin is 5v tolerant and it was only 3.3v even though it was from a different source, All I can think of is the multimeter tried to find the ground path and that was through computer USB ground and through mains power ground and back up benchtop power supply that is powering the Odrive. Or the 3.3v potential was over 10ma when testing through the multimeter. I have done the same measurements in the past with multimeter only difference is the common ground was always connected. What are your thoughts? I still would have thought there would be some noise on the PC, so possibly damaged the FTDI IC along with the microcontroller IC? Will a damaged FTDI IC stop microcontroller from working, as the 2x LEDs are from the microcontroller pins from what I can see on the board and they don't flash at all which indicates to me both ICs are dead along with faulty voltage reg that could have damaged other components.

Thanks,
Matthew

User avatar
TomKerekes
Posts: 2529
Joined: Mon Dec 04, 2017 1:49 am

Re: Tuning 2x PID Loops

Post by TomKerekes » Thu Jun 18, 2020 3:50 pm

Hi Matthew,

Sorry to hear that. If KFLOP's LEDs don't flash at all with everything disconnected except +5V then KFLOP is most likely damaged. We don't find it cost effective to attempt repairs.

Its not clear why a missing GND or measuring with a meter would have caused the damage. Was your meter set to measure current or voltage? Or maybe your meter probe shorted pins? Did the ODrive still have a GND? It seems the Motor supply is not isolated from the control signals.



Regarding the PWM: It seems there might be issues. I found this in some comments from just a few days ago which sounds bad:
The timeout for jumping from >0% duty cycle to 0% duty cycle is actually PWM_INACTIVE_TIME, set to 45ms, so it's worse than you thought. 45ms works for RC PWM, but for duty-cycle PWM it should be set relative to the nominal frequency like you said.
Maybe that could be avoided by never setting a PWM of 0. But then its not clear if short pulses can even be detected by the software approach that seems to be used. Or again by using a mode where 50% duty cycle is zero drive. Not sure what frequency you are using. There will be several cycles of delay before the drive responds and time delay is the enemy of stability. Think of a drunk driver.
Regards,

Tom Kerekes
Dynomotion, Inc.

Matt303
Posts: 8
Joined: Wed Jun 17, 2020 7:20 am

Re: Tuning 2x PID Loops

Post by Matt303 » Thu Jun 18, 2020 9:21 pm

Hi Tom,

The testing with meter was set on volts and I tested to the terminals on the Odrive, not even on the Kflop, so there was no way to short any pins. I feel something larger has happened. The 1.12V voltage regulator is reading 2.35V so that looks to be damaged and could have damaged other components? I'm based in New Zealand and it's 500 NZ dollars and I would like to attempt to repair it. I have the gear to replace the FTDI IC and FPGA IC, would the 1.2V at 2.35 damaged other stuff on the board? The Odrive still had a ground, If it was shorted it would have damaged the Odrive, but the Odrive is fine which is weird and has no protection and connected directly to the microcontroller pin.

I was planning to purchase another Kflop anyway for another project, so it looks like I will be ordering 2x and would like to at least try to repair this one. Have you got schematics I could go through to determine what else could be damaged on the board? Are the LEDs just for USB data indication or other stuff as well?

As for the PWM, I can't do any more testing at this stage but will look into the Odrive and what can be improved for the PWM response. I see there control loop is only 8khz so this could also be to slow? I did get going pretty close with 1khz PWM signal from the Kflop and just P gains set on the Kflop, I did have it holding about 1 count and had it now following motion profile, but with vibration which I was working on to tune out, and then disaster hit with the Kflop.

UPDATE: There is some life in it, I decided to plug it into a 2A USB Charger and the lights start flashing and then went solid and stayed on. If I plug it back into my PC I get nothing, tried another PC, and the same thing. So whatever has happened is making it draw to much power for USB ports on the PC. What does the LEDs solid indicate?

I have now connected an external 3A 5V Power supply to the Kflop and when I connect USB to the PC it now works and connects. I have noticed if the board is cold been sitting for a bit it takes about 30 - 60 secs until lights come on, but if I unplug and plug it in straight away the lights come on in about 5 secs so something is definitely wrong, just a matter of tracking it down.

It disconnected from PC randomly and won't reconnect. Now the 2.5V reg is reading the same as the 3.3v reg and the 1.2v reg is reading correctly at 1.19V I then pulled the power and reconnected and now it won't light up at all and back to being dead. Maybe change out the regs or possibly the FPGA IC is dead meat from the wrong voltages from the regs.

I really don't think this issue was caused by the multimeter and just happened to be bad timing when testing the encoder. If the damage was done from multimeter I would only expect that one pin to be damaged on the FPGA, very unlikely it's caused other damage around the board. Is the FPGA IC really sensitive?

Thanks,
Matthew
Last edited by Matt303 on Thu Jun 18, 2020 11:47 pm, edited 1 time in total.

User avatar
TomKerekes
Posts: 2529
Joined: Mon Dec 04, 2017 1:49 am

Re: Tuning 2x PID Loops

Post by TomKerekes » Thu Jun 18, 2020 11:47 pm

i Matthew,

The 1.2V supply is for the DSP Core. I'm surprised double voltage didn't damage it.

For the LEDs to Flash requires much of KFLOP to be working. The LEDs are connected to the FPGA and only used for status. The DSP must boot (which requires FLASH and SDRAM working), the FPGA configuration must successfuly load, and DSP-FPGA Communication. So maybe only some FPGA IO Pins damaged. Possibly the FTDI USB interface. Possibly some DSP IO that enables the USB.

If KFLOP is drawing high current I would expect something to be getting hot. That could be a clue.

The part numbers for:
U5 - TMS320C6722BRFP200
U6 - XC3S100E-4TQG144C
U10 - FT245RL


I see there control loop is only 8khz so this could also be to slow? I did get going pretty close with 1khz PWM signal from the Kflop and just P gains set on the Kflop
8KHz should be fast enough.

Good luck
Regards,

Tom Kerekes
Dynomotion, Inc.

Matt303
Posts: 8
Joined: Wed Jun 17, 2020 7:20 am

Re: Tuning 2x PID Loops

Post by Matt303 » Thu Jun 18, 2020 11:57 pm

Hi Tom,

It disconnected from PC randomly and won't reconnect. Now the 2.5V reg is reading the same as the 3.3v reg and the 1.2v reg is reading correctly at 1.19V I then pulled the power and reconnected and now it won't light up at all and back to being dead. Maybe change out the regs or possibly the FPGA IC is dead meat from the wrong voltages from the regs.

I really don't think this issue was caused by the multimeter and just happened to be bad timing when testing the encoder. If the damage was done from multimeter I would only expect that one pin to be damaged on the FPGA, very unlikely it's caused other damage around the board. Is the FPGA IC really sensitive?

Nothing on the board is getting hot which is odd. It almost seems to be a voltage reg has failed and damaged the other components, when they fail they normally spit out the supply voltage out the output. What is the 2.5v reg being used for?

Thanks,
Matthew

User avatar
TomKerekes
Posts: 2529
Joined: Mon Dec 04, 2017 1:49 am

Re: Tuning 2x PID Loops

Post by TomKerekes » Fri Jun 19, 2020 3:57 pm

Hi Matthew,

KFLOP and its FPGA are very robust. I'm not aware of an instance of failure without something improper being done to it. Its unlikely a voltage regulator failed.

A Voltmeter in Voltage mode should be very high impedance and never cause any damage.

The 2.4V is used for the FPGA Core.
Regards,

Tom Kerekes
Dynomotion, Inc.

Matt303
Posts: 8
Joined: Wed Jun 17, 2020 7:20 am

Re: Tuning 2x PID Loops

Post by Matt303 » Mon Jun 22, 2020 6:18 am

Hi Tom,

Is there an alternative TMS320C6722BRFP200 since there are not any available at Digikey, I see mouser has them available, will order from there.

How else could there be 3.2v on the 2.5v reg output if the reg has not failed? Anything else I can test to determine what has failed? Internal layer traces failed possibly? Is the 2.5v tolerant of 3.3v?

Thanks,
Matthew

Post Reply