Flattening data

[J_Dev]

Hi All, I need to flatten the data being sent using the data to server functionality specifically what I need to do is remove the outside array of the bellow object.

{
“TESTING_MQTT_DATA”: [
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “6.224992”,
“name”: “TEST_Pressure_measured_kg_m3”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.026857”,
“name”: “TEST_Pressure_NORM_20C”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “20.376083”,
“name”: “TEST_Temperature_T_Measured_degrees_c”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “11.566125”,
“name”: “TEST_Dew_Frost_Point_Temperature_Td_f_Measured_dregees_c”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “11.344722”,
“name”: “TEST_Dew_Frost_Point_Temperature_Atmospheric_Measured_degrees_c”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “13432.592773”,
“name”: “TEST_H2O_Water_Content_measured_PPM”
}
]
}

what I need to send is this

{

{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "1.028216",
  "name": "TEST_PRESSURE_Bara"
},
{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "6.224992",
  "name": "TEST_Pressure_measured_kg_m3"
},
{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "1.026857",
  "name": "TEST_Pressure_NORM_20C"
},
{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "20.376083",
  "name": "TEST_Temperature_T_Measured_degrees_c"
},
{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "11.566125",
  "name": "TEST_Dew_Frost_Point_Temperature_Td_f_Measured_dregees_c"
},
{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "11.344722",
  "name": "TEST_Dew_Frost_Point_Temperature_Atmospheric_Measured_degrees_c"
},
{
  "date": "03\u002F02\u002F2026 10:53:12",
  "data": "13432.592773",
  "name": "TEST_H2O_Water_Content_measured_PPM"
}

}

I have tried using a custom script with no affect.

Any help is greately appreciated

[Matas]

Hello,

I wish we could help you, but we are unable to provide support for custom script writing and similar work. My suggestion is to either wait for someone else to respond or use AI tools or programming-related forums to get help writing a custom script.

A note, though - “Custom” is not a Custom script, you’re just formatting the output essentially. A custom script would be you selecting “LUA Script” from the drop-down menu & then uploading the custom script file, which will get executed.

Regards,
M.

[J_Dev]

Hi Matas,
Thanks for taking the time to reply!

Actually my main query was in regards to the flattening of the data as to be able to send just the modbus registers of the device we are polling and not have the collection name wrapping the data object.

my original message was not so clear.
the project architecture is as follows
“device” → trb245 (serial client to poll the “device” and data to server to send the readings from the “Device” ) → Third party IoT platform (that expects a non nested i.e flat data structure).

sending the information using the data to server functionality we have to give the “collection” a name. and we pass the selected parameters into an array of objects (NOTE I’m sure you know this I am just giving context! ). I need to remove the collection name from the object and just have json with the selected parameters .
e.g instead of

{
“TESTING_MQTT_DATA”: [
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
},
{
“date”: “SOME DATE”,
“data”: “SOME DATA”,
“name”: “A NAME”},
{
“etc”: “etc”,
“etc”: “etc”,
“etc”: “etc”,

}

]

}

I need to remove the TESTING_MQTT_DATA array or parent object.

Something like

{
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
},
{
“date”: “SOME DATE”,
“data”: “SOME DATA”,
“name”: “A NAME”},
{
“etc”: “etc”,
“etc”: “etc”,
“etc”: “etc”,

}
}

Any suggestions on how to achieve this?

Thanks for the tip with the custom selection.

All the best,
J Dev

[Matas]

Hi there,

I believe we had a very similar query to yours not too long ago, and unfortunately, this isn’t something that is achievable by changing settings around in the WebUI. I will, however, post a suggestion for our R&D team to see if this is something they can add as a setting for the near future.

As for now, your best shot with this would be to utilize aformentioned custom scripts to try & remove that wrapper.

In case I’ll have any updates from our R&D, I’ll let you know.

Regards,
M.

[Matas]

Hello,

After further research and suggestions from my colleagues, here’s what you need to do:
First, edit the collection configuration:

This shall stay the same as it was, just ensure that you have all the data as you want:

Next, edit the Collection Actions:

You can then reformat the string that is hindering the output to the following:

%TESTING_MQTT_DATA%

This should completely remove the line you didn’t want and should only give you the outputs that you’ve setup in the previous window.

Regards,
M.

[mesrik]

J_Dev,

This is just side note(s) of your source json example is not valid json. The problem is that it contains wrong double quotation marks are not ASCII 32 (decimal) but instead appear to be UTF-8 various alternative quotation marks. And because json only ASCII quotation is allowed, that json you posted does not pass test.

Having your source.json in file on a linux box, here is what happens:

$ python3 -mjson.tool < source.json
Expecting property name enclosed in double quotes: line 2 column 1 (char 2)
$

Now that gives a hint what is wrong, but let’s see a bit another way using bash quick function found from one stackoverflow post.

$ nonascii() { LANG=C grep --color=always ‘[^ -~]+’; } # this adds function for this session

$ nonascii < source.json
“TESTING_MQTT_DATA”: [
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “6.224992”,
“name”: “TEST_Pressure_measured_kg_m3”
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.026857”,
“name”: “TEST_Pressure_NORM_20C”
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “20.376083”,
“name”: “TEST_Temperature_T_Measured_degrees_c”
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “11.566125”,
“name”: “TEST_Dew_Frost_Point_Temperature_Td_f_Measured_dregees_c”
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “11.344722”,
“name”: “TEST_Dew_Frost_Point_Temperature_Atmospheric_Measured_degrees_c”
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “13432.592773”,
“name”: “TEST_H2O_Water_Content_measured_PPM”
$

Here in this chat you don’t see that all those quoatation marks are red highlighted and non ASCII characters. Now if you try to fix it, provided that this chat won’t mess up these here. But below I copied above a bit before and after string left and right leaning quotation marks to that sed first arguments and second is ASCII 32 quotation mark.

$ sed ‘s/[“”]/"/g’ source.json > fixed-source.json
$ python3 -mjson.tool < fixed-source.json
{
“TESTING_MQTT_DATA”: [
{
“date”: “03/02/2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
},
{
“date”: “03/02/2026 10:53:12”,
“data”: “6.224992”,
“name”: “TEST_Pressure_measured_kg_m3”
},
{
“date”: “03/02/2026 10:53:12”,
“data”: “1.026857”,
“name”: “TEST_Pressure_NORM_20C”
},
{
“date”: “03/02/2026 10:53:12”,
“data”: “20.376083”,
“name”: “TEST_Temperature_T_Measured_degrees_c”
},
{
“date”: “03/02/2026 10:53:12”,
“data”: “11.566125”,
“name”: “TEST_Dew_Frost_Point_Temperature_Td_f_Measured_dregees_c”
},
{
“date”: “03/02/2026 10:53:12”,
“data”: “11.344722”,
“name”: “TEST_Dew_Frost_Point_Temperature_Atmospheric_Measured_degrees_c”
},
{
“date”: “03/02/2026 10:53:12”,
“data”: “13432.592773”,
“name”: “TEST_H2O_Water_Content_measured_PPM”
}
]
}

Now it’s valid json what we have. While getting this far, now I get to what I was trying first place show you how you can a bit cheat and ask LLM like Google search AI-mode how to modify JSON files when needed. Provided you have a manipulation tool available you can use of course. Fortunately in linux we have the “jq” with is a swiss army knife of json manipulator. It’s syntax is quite quirky but with a little bit of help from a LLM you usually find a nice solution.

So took that source.json (fixed) in editor and wrote in it

How can I change using jq transfer source json
{
“TESTING_MQTT_DATA”: [
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “6.224992”,
“name”: “TEST_Pressure_measured_kg_m3”
}
]
}

to following ?

{
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
},
{
“date”: “03\u002F02\u002F2026 10:53:12”,
“data”: “6.224992”,
“name”: “TEST_Pressure_measured_kg_m3”
}
}

Then copied it to Google Search AI-mode prompt and it gave first sort answer:

bash

$ jq '.TESTING_MQTT_DATA[]'  file.json

Looks good, let’s test how it goes:

$ jq ‘.TESTING_MQTT_DATA’ fixed-source.json
{
“date”: “03/02/2026 10:53:12”,
“data”: “1.028216”,
“name”: “TEST_PRESSURE_Bara”
}
{
“date”: “03/02/2026 10:53:12”,
“data”: “6.224992”,
“name”: “TEST_Pressure_measured_kg_m3”
}
{
“date”: “03/02/2026 10:53:12”,
“data”: “1.026857”,
“name”: “TEST_Pressure_NORM_20C”
}
{
“date”: “03/02/2026 10:53:12”,
“data”: “20.376083”,
“name”: “TEST_Temperature_T_Measured_degrees_c”
}
{
“date”: “03/02/2026 10:53:12”,
“data”: “11.566125”,
“name”: “TEST_Dew_Frost_Point_Temperature_Td_f_Measured_dregees_c”
}
{
“date”: “03/02/2026 10:53:12”,
“data”: “11.344722”,
“name”: “TEST_Dew_Frost_Point_Temperature_Atmospheric_Measured_degrees_c”
}
{
“date”: “03/02/2026 10:53:12”,
“data”: “13432.592773”,
“name”: “TEST_H2O_Water_Content_measured_PPM”
}

$ jq ‘.TESTING_MQTT_DATA’ fixed-source.json > output.json
$ python3 -mjson.tool < output.json
$

No errors, it seems to work fine. And for the bonus the Google LLM AI also gave nice explanation what that jq command line ‘script’ did its work.

OK, so even though this was bit more involved and the “jq” utility is not available at least as a package for my rutx50 I thought it would be useful perhaps for someone here that I bring forward following things:

1. Google LLM AI-mode seems be able to give quite useful code snippets how you can manipulate json and other common file formats using tools like jq. If you ever need to manipulate xml there is bit similar tool xmlstarlet which is very good in that.

2. JSON is very picky about file format and UTF-8 quotation mark problems are quite common, it’s good to know about these issues and how to fix if you encounter problems. I’ve found that most problems you get in with this is when used some general text editor which is not explicitly meant for programming. Apple iOS and macOS devices this is has happened to me more than once. Less with Linux editors. Windows I dunno, as I don’t use it.

3. And of course it’s good to know how to check your json is valid too, because if you don’t then you sooner or later end up wasting plenty of time figuring out what’s wrong with what you are doing and fight with hard to see with your plain eyes those double quote problems etc.

So once more, I wrote this in intent that information here perhaps would be useful for someone. And to someone If you already knew these, great – good for you.

Cheers,

riku

ps. Sorry about post became so long, didn’t intend but fixing that json first was important, so that jq would work with the file without spewing error.

e: typo and fixed reference to sed instead tr which I first had used and forgot to change before posting.

[J_Dev]

Hi Mesrik,

Very interesting!
Thanks for the reply.

all the best