What would cause direct FPGA I/O to break?

Moderators: TomKerekes, dynomotion

Post Reply
SJHardy
Posts: 8
Joined: Thu Oct 03, 2019 12:36 am

What would cause direct FPGA I/O to break?

Post by SJHardy » Sat Nov 30, 2019 6:35 am

Hi Tom,
I have a bit of a head-scratcher; you might have some suggestions. Here's the scenario:

We program in a supervisor thread (thread 7, using more than 64k, less than 128k, using CL6X compiler toolchain). The supervisor talks to an I/O concentrator on our motherboard using SPI bit-banged using 4 I/O pins from the kflop. All was working fine until I added some more code. What now happens is that during initialization, something is happening to break the FPGA writes so that the output bit stays low. Of course I suspected a SNAFU in my code, but after careful checking, using the hook facility in CL6X to basically "single step" each function entry, it seems that the breakage happens seemingly instantly, in the entry to an init function (not even any parameters), without executing any actual code. Sorry, I don't know how to break it down to the assembler level.

My question is, what could I be doing wrong to suddenly make the following code fail to toggle I/O #45?

Code: Select all

#define FPGAREF(X) (FPGA_ADDR + ((X)*4+2))
#define ss_fpgaset FPGAREF(BIT_SET+5)
#define ss_fpgaclr FPGAREF(BIT_CLR+5)
#define ss_hi  0x20
#define ss_lo  0xDF
static int ss_toggle = 0;

...

    while (1) {
        WaitNextTimeSlice();
         if (ss_toggle) {
            ss_toggle = 0;
            *ss_fpgaset = ss_hi;    // SS inactive (high)
        }
        else {
            ss_toggle = 1;
            *ss_fpgaclr = ss_lo;    // SS active (low)
        }
    }

The above code is actually run in the function entry hook, so rather than while(1) it actually waits for a different I/O to signal it to move on, but this allows me to scope I/O #45 to make sure it is still able to toggle at the start of each function. After inserting some dummy function calls, it turns out that toggling stops on entry to a function called i2c_init() which actually doesn't have anything to do with I/O on this board, but it isn't executed anyway since the hook is at entry.

Here's some of the linker map. Maybe there's a clue in there, but I can't see it.

Code: Select all

******************************************************************************
               TMS320C6x Linker Unix v7.4.14                   
******************************************************************************
>> Linked Fri Nov 29 21:15:28 2019

OUTPUT FILE NAME:   </home/steve/DM6/Projects/DummyQ350/Build/DM6-SCL-supervisor(7).out>
ENTRY POINT SYMBOL: "_main"  address: 800ce3e4


MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  IRAM                  1001c000   00004000  00000000  00004000  RWIX
  THREAD_MEM            800b0000   00050000  000250a4  0002af5c  RWIX
  SDRAM                 80100000   00f00000  00000000  00f00000  RWIX


SECTION ALLOCATION MAP

 output                                  attributes/
section   page    origin      length       input sections
--------  ----  ----------  ----------   ----------------
.text      0    800b0000    0001f5c0     
                  800b0000    0001f5c0     DM6-SCL-supervisor.c.o (.text)

.const     0    800cf5c0    00003500     
                  800cf5c0    00001e19     DM6-SCL-supervisor.c.o (.const:.string)
                  800d13d9    00000007     --HOLE-- [fill = 0]
                  800d13e0    000010e0     DM6-SCL-supervisor.c.o (.const:_settings_hashvars)
                  800d24c0    00000210     DM6-SCL-supervisor.c.o (.const:_i2c_static_devices)
                  800d26d0    00000078     DM6-SCL-supervisor.c.o (.const:_block_basehashvar)
                  800d2748    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_mpg_device_high)
                  800d27a8    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_mpg_device_low)
                  800d2808    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_tc_device_high)
                  800d2868    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_tc_device_low)
                  800d28c8    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_fp_device_high)
                  800d2910    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_fp_device_low)
                  800d2958    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_gpio_device_high)
                  800d29a0    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_gpio_device_low)
                  800d29e8    00000044     DM6-SCL-supervisor.c.o (.const:.string:_calibration_var_bitmap)
                  800d2a2c    00000030     DM6-SCL-supervisor.c.o (.const:_dyn_tc_ins_tab)
                  800d2a5c    0000001c     DM6-SCL-supervisor.c.o (.const)
                  800d2a78    00000018     DM6-SCL-supervisor.c.o (.const:__enab_out)
                  800d2a90    00000018     DM6-SCL-supervisor.c.o (.const:__lim_sw)
                  800d2aa8    00000018     DM6-SCL-supervisor.c.o (.const:__ok_in)

.far       0    800d2ac0    000024e8     UNINITIALIZED
                  800d2ac0    000024e8     DM6-SCL-supervisor.c.o (.far)

.switch    0    800d4fa8    000000f4     
                  800d4fa8    00000060     DM6-SCL-supervisor.c.o (.switch:_get_gp_input)
                  800d5008    00000058     DM6-SCL-supervisor.c.o (.switch:_get_solenoid)
                  800d5060    00000014     DM6-SCL-supervisor.c.o (.switch:_do_check_estop)
                  800d5074    00000014     DM6-SCL-supervisor.c.o (.switch:_spi_sm)
                  800d5088    00000014     DM6-SCL-supervisor.c.o (.switch:_translate_dti)

.placeholder 
*          0    800d50a0    00000008     
                  800d50a0    00000008     --HOLE-- [fill = aaaaaaaa]

.cinit     0    80100000    0000046c     COPY SECTION
                  80100000    00000464     DM6-SCL-supervisor.c.o (.cinit)
                  80100464    00000004     --HOLE--
                  80100468    00000004     --HOLE-- [fill = 0]

...

(from symbols in address order...)

800c3f60   ___enable_hook_wait
800c3fd0   _init_axes   <------ last substantial function to run OK.  Sets up chan[] settings.
800c4614   __               <------ this is dummy function (only printf) called just before the following.  #45 toggles ok on entry
800c46a4   _i2c_init     <------ in entry hook, #45 stuck low.
800c7a80   _init_mpg
800c7ff4   _init_axes_servos
800c8134   _init_testmode
800c83a0   _estop_recovery
800cde80   _do_mpg
800ce378   _fail_start
800ce3e4   _main

Any help would be appreciated. The code works fine if I remove a bit of code (it seems to be near some threshold of complexity) and has been working fine for a long time with our customers. But it has me tearing my hair out, and that's really something for a bald guy :-)

Regards,
SJH

User avatar
TomKerekes
Posts: 665
Joined: Mon Dec 04, 2017 1:49 am

Re: What would cause direct FPGA I/O to break?

Post by TomKerekes » Sat Nov 30, 2019 6:51 pm

Hi SJH,

I don't really follow much of that, but I see something that might be an issue. The link map shows cinit data being loaded into 0x80100000. I think cinit is a table of constant data that is supposed to be copied to global variables to initialze them on program startup. 0x80100000 is the beginning of SDRAM where the KFLOP heap (sysmem) is allocated. KFLOP uses the heap for a few things. Maybe the runtime library does also. So I believe loading your program may be corrupting some things KFLOP has currently allocated on the heap. Can't think of a way that might cause a direct bit toggle not to work, but then any corruption can manifest in strange ways. It seems you are using the TI Linker/Compiler? What do you have as a Linker command file?
Regards,

Tom Kerekes
Dynomotion, Inc.

SJHardy
Posts: 8
Joined: Thu Oct 03, 2019 12:36 am

Re: What would cause direct FPGA I/O to break?

Post by SJHardy » Sat Nov 30, 2019 7:26 pm

Hi Tom,
Thanks for the prompt response. As it happens, I came to the conclusion that it was something to do with .cinit section, since tracing through the code showed that some initialized tables in the code are getting corrupted. One of those was the I/O initialization, which was running into some garbage and setting non-existent I/O bits. I could tell because after this function the I/O would be jammed up.

Yes, I am using the CL6 compiler and linker. Here is the linker command file:

Code: Select all

--entry_point _main
--output_file=/home/steve/DM6/Projects/DummyQ350/Build/DM6-SCL-supervisor(7).out
--map_file=/home/steve/DM6/Projects/DummyQ350/Build/DM6-SCL-supervisor(7).out.map
--ram_model
--no_symtable
"/tmp/DM6-SCL-supervisor.c.o"
_persist = 0x10016a00;
_sin = 0x1000f91c;

[symbols omitted]

MEMORY {
IRAM: o = 0x1001c000, l = 0x00004000
THREAD_MEM: o = 0x800B0000, l = 0x00050000
SDRAM: o = 0x80100000, l = 0x00f00000
}
SECTIONS {
.placeholder: palign(8), fill = 0xaaaaaaaa {. += 4;} > THREAD_MEM
.text: > THREAD_MEM
.far: > THREAD_MEM
.const: > THREAD_MEM
.cinit: > SDRAM
.switch: > THREAD_MEM
}
So should I change the .cinit section to go to thread_mem? I'll give that a try.

Should I be using SDRAM at all? TBH, I can't remember where I got this linker setup, but it has worked for a long time, only now exposing some hidden issues.

Regards,
SJH

SJHardy
Posts: 8
Joined: Thu Oct 03, 2019 12:36 am

Re: What would cause direct FPGA I/O to break?

Post by SJHardy » Sat Nov 30, 2019 7:33 pm

OK, that fixed it!

Still would be useful to know a recommended linker command file.

Regards,
SJH

User avatar
TomKerekes
Posts: 665
Joined: Mon Dec 04, 2017 1:49 am

Re: What would cause direct FPGA I/O to break?

Post by TomKerekes » Sat Nov 30, 2019 7:53 pm

Hi SJH,
OK, that fixed it!
Great.
Still would be useful to know a recommended linker command file.
The Link file for the User Programs shouldn't reference the 0x80100000 SDRAM area at all.

I believe you are using an older Version with Linux but Version 4.35b has better support for the TI Compiler including a LinkTemplate.cmd file that is patched up by the KMotion Libraries. You might see this Video where the LinkTemplate.cmd is described at the end.

Regards,

Tom Kerekes
Dynomotion, Inc.

Post Reply