MDK Client Initial load of data sync error

[sopizaken]

Hello experts,

We have created an mdk project and connected it to our backend with thousands of data. When we deployed the app in Mobile Services Client, we are encountering a sync error after a couple of minutes (gateway timeout error) because of the bulk data:

Would you know how to load big data like this? Appreciate any inputs you may have.

Thank you!

[bill_froelich]

How many records are you expecting in the EquipmentDataSet?

[sopizaken]

Hi bill.froelich,

Around 50,000 records

[bing_fang]

Hi sopizaken,

thanks for your attachment. But there is no response in it. Please check Log Message Header and Message Body option in the Mobile Network Trace page.

Please capture it again and attach it.

Thanks.

[sopizaken]

Hi bing.fang,

Thanks for this, now i know where to see the retrieved data. I rerun the trace and saw that it is correctly returning the number of records. However in the display of the listpicker, there's still only one record displayed. I have not added any filter with the query options, i just specified the service and the entityset. I removed my rule first but it is still displaying a single record:

[sopizaken]

Hi bing.fang,

We have fixed the issue i just raised regarding the single record by adding some more key fields on the entityset. The question is, are we supposed to add the $top and $skip on the initializeOffline.action or where are we supposed to use it? Because when i tried to scroll down the list, it didn't load the next results.

[bing_fang]

Hi bill.froelich , this should be about MDK's UI issue. Please help to add comments.

Thanks.

[bill_froelich]

The query options for initialize offline typically would not use $top or $skip. Most of the time you would retrieve all the records (no query options) or use a $filter to restrict based on some criteria. This action simply populates the offline db files. Then in your application when displaying a list MDK will automatically use $top and $skip to read / page the data as needed when scrolling an Object Table or List picker for example.

You can also use them in the app if needed (typically in conjunction with filters and other criteria)

[sopizaken]

Hi bill.froelich,

Is it possible for the user to always connect online and just define the specific items to be downloaded offline once he has loaded the records he wanted to use offline? As i understood from the mdk offline template, it always calls the initializeOffline.action which includes the defining requests for the selected odata entities.

[bill_froelich]

Rachel,

I am not sure I fully understand your question but yes, you can add defining requests after the store is initially populated. What doesn't currently exist yet is a remove defining request option in MDK.

One approach would be to display an online list of items and allowing the user to mark the ones they then download offline for further processing. This assumes you have a backend service design that supports some kind of user selection storage and linking that when downloading the data. Having the backend handle the filtering for hte current user avoids issues with the defining requests on the client. Initially the equipment list (for example based on your screenshot) would be empty if the user has not selected equipment to download from on online lookup. Once the user selects what to download when they perform a delta sync the Equipment set would now return data since the user specified one or more items to download.

[sopizaken]

Hi bill.froelich,

Thank you so much for your response, this gives me other options to implement data fetching in our app. Regarding your first suggestion, how do we add defining requests after the store is initially populated? Currently, we have this equipment list data set in our defining request in the InitializeOffline.action, i have defined a query option for this to first get the 10,000 records. After this defining request, i wanted to call another defining request (thru custom code) that will get the next 10,000 records, then after this call another one until we get all the records.

[bill_froelich]

Normally you would just request all the equipment records and let Mobile Services retrieve the data. Your odata service can do paging to return a subset and allow Mobile Services to request additional data automatically until all the data is downloaded.

[bing_fang]

Hi sopizaken.

I am a little confused. I guess that after you adding $top and $skip, there is no 504 issue and this issue is that the data downloaded is not correct.

About 504 issue. I mean that your backend server need support pagination mechanism. For example. If there are too many records for Entity Customers. When accessing /Customers, the response only return a part of data and add a next link in the end of the body. Then mobile server will continue to get the content of next page until no next link in the response body. The mobile service will send many requests to fetch all content.

About you said there is only a record after add $top and $skip. Please capture the network trace in the Mobile service's cockpit and attach the network trace file which is very useful to check the root cause.

Thanks

[sopizaken]

tracelog.txtHi bing.fang,

"I guess that after you adding $top and $skip, there is no 504 issue and this issue is that the data downloaded is not correct." - yes this is correct,

Regarding the 504 issue, our backend is already supporting the pagination mechanism. That's why when we run the top and skip in gateway client, it is returning the correct number of records.

I have attached the network trace for the single record returned after adding $top=10 and $skip=1.

tracelog.txt

[bing_fang]

According to the error message, it looks like the 504 code is from the backend server. You can use pagination mechanism to return the big data to avoid wait for long time without data.

Thanks.

[sopizaken]

Hi bing.fang,

I have tried to use $top and $skip in the queryoptions on InitializeOffline.action, but it is not working. It always returns a single record, regardless of the value assigned to $top, e.g. $top=5 returns 1, $top=100 still returns 1 record. But in gateway client, i can get the correct number of records when i try to use the $top. Would you know how to fix this issue?

[bill_froelich]

I would suggest checking the logs of the backend odata server. Also check the Mobile Services logs to see if there is anything more that might be helpful. It also looks like there is more to the MDK Client error that might also be helpful so I would suggest checking the MDK client log as well.

I have loaded much more data than that into offline MDK apps so there should be no issue with loading that much data. Other than the time it might take.

[sopizaken]

Hi Bill,

Thank you for your response. May I know how you implemented the big bulk of data? Did you just load the service without any filters or increased some timeout sessions in the backend?

[bill_froelich]

In my cases I just call the service without any filters to retrieve all the records. I would recommend starting with checking the service logs to see if there are any errors.

[sopizaken]

Hi bill.froelich,

The only error in the logs is the same as above, its a timeout error.