What would cause direct FPGA I/O to break?

[SJHardy]

Hi Tom,
I have a bit of a head-scratcher; you might have some suggestions. Here's the scenario:

We program in a supervisor thread (thread 7, using more than 64k, less than 128k, using CL6X compiler toolchain). The supervisor talks to an I/O concentrator on our motherboard using SPI bit-banged using 4 I/O pins from the kflop. All was working fine until I added some more code. What now happens is that during initialization, something is happening to break the FPGA writes so that the output bit stays low. Of course I suspected a SNAFU in my code, but after careful checking, using the hook facility in CL6X to basically "single step" each function entry, it seems that the breakage happens seemingly instantly, in the entry to an init function (not even any parameters), without executing any actual code. Sorry, I don't know how to break it down to the assembler level.

My question is, what could I be doing wrong to suddenly make the following code fail to toggle I/O #45?

Code: Select all

#define FPGAREF(X) (FPGA_ADDR + ((X)*4+2))
#define ss_fpgaset FPGAREF(BIT_SET+5)
#define ss_fpgaclr FPGAREF(BIT_CLR+5)
#define ss_hi  0x20
#define ss_lo  0xDF
static int ss_toggle = 0;

...

    while (1) {
        WaitNextTimeSlice();
         if (ss_toggle) {
            ss_toggle = 0;
            *ss_fpgaset = ss_hi;    // SS inactive (high)
        }
        else {
            ss_toggle = 1;
            *ss_fpgaclr = ss_lo;    // SS active (low)
        }
    }

The above code is actually run in the function entry hook, so rather than while(1) it actually waits for a different I/O to signal it to move on, but this allows me to scope I/O #45 to make sure it is still able to toggle at the start of each function. After inserting some dummy function calls, it turns out that toggling stops on entry to a function called i2c_init() which actually doesn't have anything to do with I/O on this board, but it isn't executed anyway since the hook is at entry.

Here's some of the linker map. Maybe there's a clue in there, but I can't see it.

Code: Select all

******************************************************************************
               TMS320C6x Linker Unix v7.4.14                   
******************************************************************************
>> Linked Fri Nov 29 21:15:28 2019

OUTPUT FILE NAME:   </home/steve/DM6/Projects/DummyQ350/Build/DM6-SCL-supervisor(7).out>
ENTRY POINT SYMBOL: "_main"  address: 800ce3e4


MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  IRAM                  1001c000   00004000  00000000  00004000  RWIX
  THREAD_MEM            800b0000   00050000  000250a4  0002af5c  RWIX
  SDRAM                 80100000   00f00000  00000000  00f00000  RWIX


SECTION ALLOCATION MAP

 output                                  attributes/
section   page    origin      length       input sections
--------  ----  ----------  ----------   ----------------
.text      0    800b0000    0001f5c0     
                  800b0000    0001f5c0     DM6-SCL-supervisor.c.o (.text)

.const     0    800cf5c0    00003500     
                  800cf5c0    00001e19     DM6-SCL-supervisor.c.o (.const:.string)
                  800d13d9    00000007     --HOLE-- [fill = 0]
                  800d13e0    000010e0     DM6-SCL-supervisor.c.o (.const:_settings_hashvars)
                  800d24c0    00000210     DM6-SCL-supervisor.c.o (.const:_i2c_static_devices)
                  800d26d0    00000078     DM6-SCL-supervisor.c.o (.const:_block_basehashvar)
                  800d2748    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_mpg_device_high)
                  800d27a8    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_mpg_device_low)
                  800d2808    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_tc_device_high)
                  800d2868    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_tc_device_low)
                  800d28c8    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_fp_device_high)
                  800d2910    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_fp_device_low)
                  800d2958    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_gpio_device_high)
                  800d29a0    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_gpio_device_low)
                  800d29e8    00000044     DM6-SCL-supervisor.c.o (.const:.string:_calibration_var_bitmap)
                  800d2a2c    00000030     DM6-SCL-supervisor.c.o (.const:_dyn_tc_ins_tab)
                  800d2a5c    0000001c     DM6-SCL-supervisor.c.o (.const)
                  800d2a78    00000018     DM6-SCL-supervisor.c.o (.const:__enab_out)
                  800d2a90    00000018     DM6-SCL-supervisor.c.o (.const:__lim_sw)
                  800d2aa8    00000018     DM6-SCL-supervisor.c.o (.const:__ok_in)

.far       0    800d2ac0    000024e8     UNINITIALIZED
                  800d2ac0    000024e8     DM6-SCL-supervisor.c.o (.far)

.switch    0    800d4fa8    000000f4     
                  800d4fa8    00000060     DM6-SCL-supervisor.c.o (.switch:_get_gp_input)
                  800d5008    00000058     DM6-SCL-supervisor.c.o (.switch:_get_solenoid)
                  800d5060    00000014     DM6-SCL-supervisor.c.o (.switch:_do_check_estop)
                  800d5074    00000014     DM6-SCL-supervisor.c.o (.switch:_spi_sm)
                  800d5088    00000014     DM6-SCL-supervisor.c.o (.switch:_translate_dti)

.placeholder 
*          0    800d50a0    00000008     
                  800d50a0    00000008     --HOLE-- [fill = aaaaaaaa]

.cinit     0    80100000    0000046c     COPY SECTION
                  80100000    00000464     DM6-SCL-supervisor.c.o (.cinit)
                  80100464    00000004     --HOLE--
                  80100468    00000004     --HOLE-- [fill = 0]

...

(from symbols in address order...)

800c3f60   ___enable_hook_wait
800c3fd0   _init_axes   <------ last substantial function to run OK.  Sets up chan[] settings.
800c4614   __               <------ this is dummy function (only printf) called just before the following.  #45 toggles ok on entry
800c46a4   _i2c_init     <------ in entry hook, #45 stuck low.
800c7a80   _init_mpg
800c7ff4   _init_axes_servos
800c8134   _init_testmode
800c83a0   _estop_recovery
800cde80   _do_mpg
800ce378   _fail_start
800ce3e4   _main

Any help would be appreciated. The code works fine if I remove a bit of code (it seems to be near some threshold of complexity) and has been working fine for a long time with our customers. But it has me tearing my hair out, and that's really something for a bald guy

Regards,
SJH

[TomKerekes]

Hi SJH,

I don't really follow much of that, but I see something that might be an issue. The link map shows cinit data being loaded into 0x80100000. I think cinit is a table of constant data that is supposed to be copied to global variables to initialze them on program startup. 0x80100000 is the beginning of SDRAM where the KFLOP heap (sysmem) is allocated. KFLOP uses the heap for a few things. Maybe the runtime library does also. So I believe loading your program may be corrupting some things KFLOP has currently allocated on the heap. Can't think of a way that might cause a direct bit toggle not to work, but then any corruption can manifest in strange ways. It seems you are using the TI Linker/Compiler? What do you have as a Linker command file?

[SJHardy]

Hi Tom,
Thanks for the prompt response. As it happens, I came to the conclusion that it was something to do with .cinit section, since tracing through the code showed that some initialized tables in the code are getting corrupted. One of those was the I/O initialization, which was running into some garbage and setting non-existent I/O bits. I could tell because after this function the I/O would be jammed up.

Yes, I am using the CL6 compiler and linker. Here is the linker command file:

Code: Select all

--entry_point _main
--output_file=/home/steve/DM6/Projects/DummyQ350/Build/DM6-SCL-supervisor(7).out
--map_file=/home/steve/DM6/Projects/DummyQ350/Build/DM6-SCL-supervisor(7).out.map
--ram_model
--no_symtable
"/tmp/DM6-SCL-supervisor.c.o"
_persist = 0x10016a00;
_sin = 0x1000f91c;

[symbols omitted]

MEMORY {
IRAM: o = 0x1001c000, l = 0x00004000
THREAD_MEM: o = 0x800B0000, l = 0x00050000
SDRAM: o = 0x80100000, l = 0x00f00000
}
SECTIONS {
.placeholder: palign(8), fill = 0xaaaaaaaa {. += 4;} > THREAD_MEM
.text: > THREAD_MEM
.far: > THREAD_MEM
.const: > THREAD_MEM
.cinit: > SDRAM
.switch: > THREAD_MEM
}

So should I change the .cinit section to go to thread_mem? I'll give that a try.

Should I be using SDRAM at all? TBH, I can't remember where I got this linker setup, but it has worked for a long time, only now exposing some hidden issues.

Regards,
SJH

[SJHardy]

OK, that fixed it!

Still would be useful to know a recommended linker command file.

Regards,
SJH

[TomKerekes]

Hi SJH,

OK, that fixed it!

Great.

Still would be useful to know a recommended linker command file.

The Link file for the User Programs shouldn't reference the 0x80100000 SDRAM area at all.

I believe you are using an older Version with Linux but Version 4.35b has better support for the TI Compiler including a LinkTemplate.cmd file that is patched up by the KMotion Libraries. You might see this Video where the LinkTemplate.cmd is described at the end.

[SJHardy]

Well I thought I had it fixed, but there is some other link/load problem lurking in there.

It seems that when our main program (thread 7) tries to load past about 0x800D0000 (i.e. bigger than 128k program size) the LoadCoff() function is encountering a timeout. This is before the program gets executed. I put some tracing in the loader to see what was going on, and also ran a diff of the link map output between a program which works, and one which doesn't. When the loader fails, everything fails thereafter so we have to power cycle the Kflop.

Here's the linker map for a program which fails. It only uses the THREAD_MEM area:

Code: Select all

OUTPUT FILE NAME:   </home/steve/DM6/Projects/Q350/Build/DM6-SCL-supervisor(7).out>
ENTRY POINT SYMBOL: "_main"  address: 800ca428


MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  IRAM                  1001c000   00004000  00000000  00004000  RWIX
  THREAD_MEM            800b0000   00050000  0002091c  0002f6e4  RWIX
  SDRAM                 80100000   00f00000  00000000  00f00000  RWIX


SECTION ALLOCATION MAP

 output                                  attributes/
section   page    origin      length       input sections
--------  ----  ----------  ----------   ----------------
.cinit     0    800b0000    00000254     COPY SECTION
                  800b0000    0000024c     DM6-SCL-supervisor.c.o (.cinit)
                  800b024c    00000004     --HOLE--
                  800b0250    00000004     --HOLE-- [fill = 0]

.text      0    800b0000    0001b260     
                  800b0000    0001b260     DM6-SCL-supervisor.c.o (.text)

.const     0    800cb260    00003038     
                  800cb260    0000179b     DM6-SCL-supervisor.c.o (.const:.string)
                  800cc9fb    00000005     --HOLE-- [fill = 0]
                  800cca00    000011d0     DM6-SCL-supervisor.c.o (.const:_settings_hashvars)
                  800cdbd0    00000210     DM6-SCL-supervisor.c.o (.const:_i2c_static_devices)
                  800cdde0    000000bc     DM6-SCL-supervisor.c.o (.const:_iotable)
                  800cde9c    00000080     DM6-SCL-supervisor.c.o (.const:_block_basehashvar)
                  800cdf1c    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_mpg_device_high)
                  800cdf7c    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_mpg_device_low)
                  800cdfdc    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_tc_device_high)
                  800ce03c    00000060     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_tc_device_low)
                  800ce09c    00000048     DM6-SCL-supervisor.c.o (.const:.string:_calibration_var_bitmap)
                  800ce0e4    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_fp_device_high)
                  800ce12c    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_fp_device_low)
                  800ce174    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_gpio_device_high)
                  800ce1bc    00000048     DM6-SCL-supervisor.c.o (.const:_i2c_dyn_gpio_device_low)
                  800ce204    00000030     DM6-SCL-supervisor.c.o (.const:_dyn_tc_ins_tab)
                  800ce234    0000001c     DM6-SCL-supervisor.c.o (.const)
                  800ce250    00000018     DM6-SCL-supervisor.c.o (.const:__enab_out)
                  800ce268    00000018     DM6-SCL-supervisor.c.o (.const:__lim_sw)
                  800ce280    00000018     DM6-SCL-supervisor.c.o (.const:__ok_in)

.far       0    800ce298    00002538     UNINITIALIZED
                  800ce298    00002538     DM6-SCL-supervisor.c.o (.far)

.switch    0    800d07d0    00000144     
                  800d07d0    00000060     DM6-SCL-supervisor.c.o (.switch:_get_gp_input)
                  800d0830    00000058     DM6-SCL-supervisor.c.o (.switch:_get_solenoid)
                  800d0888    00000038     DM6-SCL-supervisor.c.o (.switch:_echain_msg)
                  800d08c0    0000002c     DM6-SCL-supervisor.c.o (.switch:_do_check_estop)
                  800d08ec    00000014     DM6-SCL-supervisor.c.o (.switch:_spi_sm)
                  800d0900    00000014     DM6-SCL-supervisor.c.o (.switch:_translate_dti)

.placeholder 
*          0    800d0918    00000008     
                  800d0918    00000008     --HOLE-- [fill = aaaaaaaa]

Here is some of the debug print from LoadCoff (basically printing the script commands)...

Code: Select all

CKMotionDLL::LoadCoff: /home/steve/CNCData/profiles/Q350/programs/DM6-SCL-supervisor(7).out to thread 7; PackToFlash=0
LoadCoff: LOADDATA 800D0918 8
LoadCoff: AA AA AA AA AA AA AA AA
LoadCoff: LOADDATA 800B0000 4000
LoadCoff: 58 00 98 00 90 08 00 90 58 10 90 01 14 36 8C 83 5A 10 18 00 00 20 00 00 A0 09 93 02 28 4E 71 03 F9 ED 94 02 69 06 40 03 5A E0 03 00 65 AA 98 02 10 FF FF 2F 14 36 8C 23 A2 1C 11 02 00 20 00 00
(etc.)

LoadCoff: LOADDATA 800B4000 4000
LoadCoff: 58 10 10 02 75 02 04 02 F8 BD 14 04 74 22 04 04 74 02 8C 05 64 03 28 03 F8 04 00 02 E6 82 3D 02 00 20 00 00 20 CA 90 00 00 00 00 00 2B 10 80 92 10 0A 00 90 7A 8A 14 00 FA 00 00 80 10 19 00 20
SocketWrapper::Read: timed out (TCP/IP recv)

so you can see the first 16k block transfers OK, but the next fails on first line after the LOADDATA.

I suspect it's the initial load of the .placeholder section above the magic threshold which is doing something, because a program that weighs in under 128k total works fine.

Can you give me any pointers as to how to diagnose/fix this issue?

Regards,
SJH

[TomKerekes]

Hi SJH,

I think I understand most of that but can't think of any reason it should occur.

What Version of Firmware is in KFLOP?

Is anything already running in Thread #7 as it is being loaded? Or are any other Threads running? (shouldn't be a problem unless they are somehow referencing Thread #7).

I tested this program that COFF loads more that 256K bytes using Version 4.35b and the TI Compiler under Windows and it works ok.

Code: Select all

#pragma TI_COMPILER
#include "KMotionDef.h"

#define VAL_1X     42.0
#define VAL_2X     VAL_1X,  VAL_1X
#define VAL_4X     VAL_2X,  VAL_2X
#define VAL_8X     VAL_4X,  VAL_4X
#define VAL_16X    VAL_8X,  VAL_8X
#define VAL_32X    VAL_16X, VAL_16X
#define VAL_64X    VAL_32X, VAL_32X
#define VAL_128X   VAL_64X, VAL_64X
#define VAL_256X   VAL_128X, VAL_128X
#define VAL_512X   VAL_256X, VAL_256X
#define VAL_1024X  VAL_512X, VAL_512X
#define VAL_2048X  VAL_1024X, VAL_1024X
#define VAL_4096X  VAL_2048X, VAL_2048X
#define VAL_8192X  VAL_4096X, VAL_4096X
#define VAL_16384X VAL_8192X, VAL_8192X
#define VAL_32768X VAL_16384X, VAL_16384X

const double big[32768]={VAL_32768X};

main()
{
	printf("Val = %f\n",big[32700]);  // send message to console
}

Here is the beginning of the Map File:

Code: Select all

******************************************************************************
               TMS320C6x Linker PC v7.4.24                     
******************************************************************************
>> Linked Sat May 30 10:21:54 2020

OUTPUT FILE NAME:   <C:\KMotionSrc\C Programs\BigThread7(7).out>
ENTRY POINT SYMBOL: "___user_init"  address: 800b0080


MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  IRAM                  1001b5b8   00004a48  00000000  00004a48  RWIX
  USER_SA               800b0000   00050000  000400ea  0000ff16  RWIX


SECTION ALLOCATION MAP

 output                                  attributes/
section   page    origin      length       input sections
--------  ----  ----------  ----------   ----------------
.mytext    0    800b0000    00000008     
                  800b0000    00000008     --HOLE-- [fill = aaaaaaaa]

.text      0    800b0020    000000c0     
                  800b0020    00000060     BigThread7.obj (.text)
                  800b0080    00000060     user_boot.obj (.text)

.stack     0    800b00e0    00000000     UNINITIALIZED

.bss       0    800b00e0    00000000     UNINITIALIZED

.const     0    800b00e0    0004000a     
                  800b00e0    00040000     BigThread7.obj (.const:_big)
                  800f00e0    0000000a     BigThread7.obj (.const:.string)

.data      0    800f00ea    00000000     UNINITIALIZED

.far       0    800f00ea    00000000     UNINITIALIZED

.switch    0    800f00ea    00000000     UNINITIALIZED

.cio       0    800f00ea    00000000     UNINITIALIZED

.cinit     0    800f00ea    00000000     UNINITIALIZED


GLOBAL SYMBOLS: SORTED ALPHABETICALLY BY Name 

I don't understand the purpose of the placeholder section. I don't think we added that. I don't think it is needed. But I don't think it should be a problem.

I guess the good news is that it fails consistently, correct?

If you post the COFF file that fails I can try it under V4.35 and Windows.

[SJHardy]

Hi Tom,

Currently using 4.33q. No thread is executing at time of loading. Fails entirely predictably.

Where is 4.35b? Web site downloads only seem to go up to 4.34. I should probably get that version and diff the COFF loader code to make sure there is not some old bug in there. Maybe we should be using a more recent firmware, although it would be a major hassle for us to update all our customers and to date we have had no issues with that firmware (if it ain't broke don't fix it).

I got rid of the placeholder section. Doing that actually allows it to load more of the code before losing comms, however when it gets past the 128k mark it crashes out as before. This time, there is a suspicious load address. Here are the last few commands sent to the Kflop:

Code: Select all

LoadCoff: LOADDATA 800CE69C 2C
LoadCoff: 60 B2 0C 80 7A B2 0C 80 B2 B2 0C 80 EF B2 0C 80 0B B3 0C 80 52 B3 0C 80 8B B3 0C 80 D5 B3 0C 80 1F B4 0C 80 69 B4 0C 80 7B B4 0C 80
LoadCoff: LOADDATA 800CE6C8 2C
LoadCoff: A6 B4 0C 80 B4 B4 0C 80 C0 B4 0C 80 CC B4 0C 80 DA B4 0C 80 E5 B4 0C 80 F0 B4 0C 80 FB B4 0C 80 06 B5 0C 80 11 B5 0C 80 1B B5 0C 80
LoadCoff: LOADDATA 800CF3EC 2
LoadCoff: 00 00
LoadCoff: LOADDATA F3F40000 1CE
LoadCoff: 0C 80 00 00 00 00 00 00 00 00 78 01 00 00 38 F4 0C 80 00 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 05 00 00 00 01 00 00 00 00 00 00 00 00 00
LoadCoff: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

That F3F40000 address looks real bad. Don't yet know whether it's the loader, or the Ti linker. We're using the C6000 tool chain version 7.4.23 (Linux version).

I think I should do my homework first, but if can't work it out then I can post the COFF file.

Regards,
SJH

[SJHardy]

Something is not right in CLOAD.CPP, cload_cinit(). I tried stepping through the code and can see where the address gets corrupted, starting at line

temp = (T_SIZE)unpack((unsigned char *)(loadbuf + i), sizeof(T_SIZE), sizeof(T_SIZE), 0);

where it seems to get a bad value (0x4000) and subsequently where it reads the address. I don't know enough about COFF to be confident in diagnosing the problem, so I am attaching the .out file for you to have a look at.

Regards,
SJH

[TomKerekes]

Hi SJH,

I tried the latest code and unpacks the bad F3F40000 address also.

On a side note the only change I see between V4.33q and V4.35b is this:

Code: Select all

          if (sym->n_scnum == 0) 
	     lookup_sym((struct syment *)index, (union auxent *)sym, (short)aux);  

changed to:

Code: Select all

          if (sym->n_scnum == 0) 
		  lookup_sym((struct syment *)index, (union auxent *)sym, 0);  

But the code isn't ever executed when downloading your .out file and I'm not sure what the change does.

I'm not very familiar with COFF files either. This code was written by TI.

Could you post your Linker cmd file and the Map file for us to check for anything odd or related.

Could you try compiling my "BigThread7" program to see if it creates the problem also?

V4.35b and test version info is in our wiki and forum.