jquery - Parsing Filter Needed For Newline Delimited JSON format -

i'm running problem attempt automate api process bigquery.

the issue need data in newline delimited json format go bigquery database data i'm pulling not that, need parse out.

here link pastebin can idea of data looks like, also, here because:

{"type":"user.list","users":[{"type":"user","id":"581c13632f25960e6e3dc89a","user_id":"ieo2e6dtsqhiyhtr","anonymous":false,"email":"test@gmail.com","name":"joe martinez","pseudonym":null,"avatar":{"type":"avatar","image_url":null},"app_id":"b5vkxvop","companies":{"type":"company.list","companies":[]},"location_data":{"type":"location_data","city_name":"houston","continent_code":"na","country_name":"united states","latitude":29.7633,"longitude":-95.3633,"postal_code":"77002","region_name":"texas","timezone":"america/chicago","country_code":"usa"},"last_request_at":1478235114,"last_seen_ip":"66.87.120.30","created_at":1478234979,"remote_created_at":1478234944,"signed_up_at":1478234944,"updated_at":1478235145,"session_count":1,"social_profiles":{"type":"social_profile.list","social_profiles":[]},"unsubscribed_from_emails":false,"user_agent_data":"mozilla/5.0 (linux; android 6.0.1; sm-g920p build/mmb29k) applewebkit/537.36 (khtml, gecko) chrome/54.0.2840.68 mobile safari/537.36","tags":{"type":"tag.list","tags":[]},"segments":{"type":"segment.list","segments":[{"type":"segment","id":"57d2ea275bfcebabd516d963"},{"type":"segment","id":"57d2ea265bfcebabd516d962"}]},"custom_attributes":{"claimcount":"1","membertype":"claimant"}},{"type":"user","id":"581c22a19a1dc02c460541df","user_id":"1o3helrdv58cxm7jf","anonymous":false,"email":"test@mail.com","name":"joe coleman","pseudonym":null,"avatar":{"type":"avatar","image_url":null},"app_id":"b5vkxvop","companies":{"type":"company.list","companies":[]},"location_data":{"type":"location_data","city_name":"san jose","continent_code":"na","country_name":"united states","latitude":37.3394,"longitude":-121.895,"postal_code":"95141","region_name":"california","timezone":"america/los_angeles","country_code":"usa"},"last_request_at":1478239113,"last_seen_ip":"216.151.183.47","created_at":1478238881,"remote_created_at":1478238744,"signed_up_at":1478238744,"updated_at":1478239113,"session_count":1,"social_profiles":{"type":"social_profile.list","social_profiles":[]},"unsubscribed_from_emails":false,"user_agent_data":"mozilla/5.0 (windows nt 6.3; wow64; rv:49.0) gecko/20100101 firefox/49.0","tags":{"type":"tag.list","tags":[]},"segments":{"type":"segment.list","segments":[{"type":"segment","id":"57d2ea275bfcebabd516d963"},{"type":"segment","id":"57d2ea265bfcebabd516d962"}]},"custom_attributes":{"claimcount":"2","membertype":"claimant"}}],"scroll_param":"24ba0fac-b8f9-46b2-944a-9bb523dcd1b1"} 

the 2 problems first line:

{"type":"user.list","users": 

and final piece @ bottom:

,"scroll_param":"24bd0rac-b2f9-46b2-944a-9zz543dcd1b1"} 

if eliminate two, left necessary data needed, , know filter needed parse out put in newline delimited format.

you can see playing around tool, if copy , paste first open bracket close bracket on final line, set "compact output" , apply filter:

.[] 

the result see here, in nice , neat newline delimited format see here., here not in link:

{"type":"user","id":"581c13632f25960e6e3dc89a","user_id":"ieo2e6dtsqhiyhtr","anonymous":false,"email":"test@gmail.com","name":"joe martinez","pseudonym":null,"avatar":{"type":"avatar","image_url":null},"app_id":"b5vkxvop","companies":{"type":"company.list","companies":[]},"location_data":{"type":"location_data","city_name":"houston","continent_code":"na","country_name":"united states","latitude":29.7633,"longitude":-95.3633,"postal_code":"77002","region_name":"texas","timezone":"america/chicago","country_code":"usa"},"last_request_at":1478235114,"last_seen_ip":"66.87.120.30","created_at":1478234979,"remote_created_at":1478234944,"signed_up_at":1478234944,"updated_at":1478235145,"session_count":1,"social_profiles":{"type":"social_profile.list","social_profiles":[]},"unsubscribed_from_emails":false,"user_agent_data":"mozilla/5.0 (linux; android 6.0.1; sm-g920p build/mmb29k) applewebkit/537.36 (khtml, gecko) chrome/54.0.2840.68 mobile safari/537.36","tags":{"type":"tag.list","tags":[]},"segments":{"type":"segment.list","segments":[{"type":"segment","id":"57d2ea275bfcebabd516d963"},{"type":"segment","id":"57d2ea265bfcebabd516d962"}]},"custom_attributes":{"claimcount":"1","membertype":"claimant"}} {"type":"user","id":"581c22a19a1dc02c460541df","user_id":"1o3helrdv58cxm7jf","anonymous":false,"email":"test@mail.com","name":"joe coleman","pseudonym":null,"avatar":{"type":"avatar","image_url":null},"app_id":"b5vkxvop","companies":{"type":"company.list","companies":[]},"location_data":{"type":"location_data","city_name":"san jose","continent_code":"na","country_name":"united states","latitude":37.3394,"longitude":-121.895,"postal_code":"95141","region_name":"california","timezone":"america/los_angeles","country_code":"usa"},"last_request_at":1478239113,"last_seen_ip":"216.151.183.47","created_at":1478238881,"remote_created_at":1478238744,"signed_up_at":1478238744,"updated_at":1478239113,"session_count":1,"social_profiles":{"type":"social_profile.list","social_profiles":[]},"unsubscribed_from_emails":false,"user_agent_data":"mozilla/5.0 (windows nt 6.3; wow64; rv:49.0) gecko/20100101 firefox/49.0","tags":{"type":"tag.list","tags":[]},"segments":{"type":"segment.list","segments":[{"type":"segment","id":"57d2ea275bfcebabd516d963"},{"type":"segment","id":"57d2ea265bfcebabd516d962"}]},"custom_attributes":{"claimcount":"2","membertype":"claimant"}} 

so need filter can apply in same manner used .[] pull out text prior first open bracket (as highlighted above) text prior closed bracket @ end.

but here's final problem comes in. while need final piece of text out of equation, still need string of letters , numbers known scroll paramater. because in order capture data need in api, need continuously use new scroll paramater generates command line call until data in.

the initial call looks such:

$ curl -s https://api.program.io/users/scroll -u 'dg9romu5ngfjytkwxzlindffngiymf9iyza0xzu0ndg3mje5zwjkzdoxoja=': -h 'accept:application/json' 

but in ordere info in, need scroll parameter seperate call looks like:

curl -s https://api.intercom.io/users/scroll?scroll_param=foo -u 'dg9romu5ngfjytkwxzlindffngiymf9iyza0xzu0ndg3mje5zwjkzdoxoja=': -h 'accept:application/json' >scroll.json 

so while need rid of text in blob contains paramater in order put in newline delimited format, still need extract whatever paramater loop script continue run until empty.

would love hear advice in working around this!

like others have posted comments, won't pretend understand details of specific question, if general question how use jq emit newline-delimited json (that is, ensure each json text followed newline, , no other (raw) newlines added), answer simple: use jq -c option, , without -r option.