How to avoid changes in package-lock.json when using local and BAS development in parallel?

[gregorw]

Hello BAS Team,

we've developed a CAP application in our local VS Code environment. Now we're handing over the applicaiton to the customer and they must use BAS as no local IDE installation is allowed. During the setup of the project in BAS I've noticed the following issue:

When running "npm install" in BAS the original content of package-lock.json e.g.:

    "@sap/cds-odata-v2-adapter-proxy": {
      "version": "1.8.2",
      "resolved": "https://registry.npmjs.org/@sap/cds-odata-v2-adapter-proxy/-/cds-odata-v2-adapter-proxy-1.8.2.tgz",
      "integrity": "sha512-fN36iWJSPIOvxnPgbKmbD4FHvXV2QxUR3cVzWLNHmVY9/C1ky1oODfsYN4fkIuxj4nuIfjYdKIsaLjnv50fRmw==",

is replaced with:

    "@sap/cds-odata-v2-adapter-proxy": {
      "version": "1.8.2",
      "resolved": "http://nginx-redirector.repo-cache.svc.cluster.local/repository/appstudio-npm-group/@sap/cds-odata-v2-adapter-proxy/-/cds-odata-v2-adapter-proxy-1.8.2.tgz",
      "integrity": "sha512-fN36iWJSPIOvxnPgbKmbD4FHvXV2QxUR3cVzWLNHmVY9/C1ky1oODfsYN4fkIuxj4nuIfjYdKIsaLjnv50fRmw==",

as you can see the version and integrity fields are exactly the same. But the resolved field does change and this results in a change that either needs to be commited to the Git repository or discarded. My temporary solution for the moment is to use:

npm ci

I understand that the registry is configured to a different location:

registry = "http://nginx-redirector.repo-cache.svc.cluster.local/repository/appstudio-npm-group/"<br>

to improve performance and avoid not necessary traffic. But maybe there is a way to keep the original resolved value?

Best Regards
Gregor

[SebastianEsch]

I noticed this too. This is quite annoying if you have mixed setups with BAS and VS Code developers.

The BAS internal resolved URL also breaks in CI/CD pipelines. From my point of view, a NPM cache should be transparent and not change your development artefacts.

Kind regards,

Sebastian

[michelle-van-lieshout]

I developed in VS before too and then wanted to change to the BAS environment to add some Fiori elements -elements. From BAS I need to deploy to a cloud foundry environment.

This change, that BAS makes in comparison to Visual Studio Code, also results in errors trying to deploy to a CF environment. Deploying to CF now is not possible from the BAS environment. I tried removing the "ignore: "*node_modules*"" from the build parameters in the mta.yaml but this didn't solve the problem. I keep getting the error: Unable to build dependencies: exit status 1

This error is due to the network requests to the local cache failing:

network request to http://nginx-redirector.repo-cache.svc.cluster.local/repository/appstudio-npm-group/yallist/-/yallis... failed, reason: getaddrinfo ENOTFOUND nginx-redirector.repo-cache.svc.cluster.local (STDOUT, STG)#

A solution to this problem would therefore be very helpful to me too! Keep me updated 🙂

[gregorw]

michelle-van-lieshout in which step does your error occur? During mbt build in BAS or during the MTA deployment?

[michelle-van-lieshout]

gregorw it occurs during the MTA deployment. I solved it for now by pulling everything from git in VS, then running all npm installs and copying everything within the package-lock.json to BAS (so you get the registry.xxxx link instead of the nginx-redirectorxxxxx links within the package-lock.json files).

[gregorw]

Can you please file an incident via SAP Support so we get pressure also via an official support channel.

[michelle-van-lieshout]

I have filed an incident as well 😉

[michelle-van-lieshout]

gregorw: The incident: 896313 / 2021

[thomas_jung · Accepted solution]

The official answer back from development is that you should use an .npmrc (in the project or user level). Point the registry configuration in the .npmrc to the normal, public NPM registry. This will override the default BAS configuration and force it to use the normal registry instead of the cache one.

[gregorw · Accepted solution]

Hi Thomas,

thank you for this follow up. I've added now a .npmrc with the following content:

registry=https://registry.npmjs.com/ 

To all folders that contain a package.json and in which an npm install is executed. That avoids changes on the package-lock.json.

Best Regards
Gregor

[thomas_jung]

I asked the BAS development team internally about this and their response was: Package-lock.json should be excluded from git. BAS resolves against NPM mirror, so these URLs will be in the package-lock.json. In any environment that does not have access to Nexus it breaks builds.

[gregorw]

Hi Thomas,

thank you for jumping in hear. According to the documenation for the package-lock-json which states:

"This file is intended to be committed into source repositories"

I have to disagree with the answer from the BAS development team to not add the file to version control. Please ask them to join the discussion here directly.

Best Regards
Gregor

[thomas_jung]

I have shared the link to this thread with them. I also had similar questions back as I also believe package-lock.json committed to git is a best practice particularly in CI/CD setup.

[gregorw]

Hi Thomas,

today I faced another issue that is realated to the package-lock.json. As I try to keep the MTAR resulting from the build as small as possible I've added:

    build-parameters:
      ignore: ["node_modules/"]

to my srv and db module. But when I now build this in BAS the deploy fails as the CF build environment seems to have no access to http://nginx-redirector.repo-cache.svc.cluster.local. I've now posted: Deployment of MTAR fails when build-parameters: ignore: ["node_modules/"] is set on this topic.

Best Regards
Gregor

[thomas_jung]

I can see where that would be an issue. You could also exclude the package-lock.json with the same build-parameters ignore configuration; but in doing so you are of course running the risk of deploying into production with a different version of modules than you developed and tested with. I'll share this additional item with development as well.

[WRoeckelein]

I agree with Gregor and Thomas that package-lock.json should be commited to git to get reproducible builds. And for reproducible builds the package-lock.json needs to contain public acessible URLs.

If you fear direct outside access to arbitrary npm repos, you should leave the package-lock.json as it is and put some other checks in between.

And please do not start the need to commit node-modules or put it into mtars. We already have enough problems with the node-modules file system space usage (I remember at least one incident by us caused by this), we should not carry this over to git and mtar (i would wish to have non-inclusion of node-modules as the default)! I would even think if its time for pnpm usage...

[SebastianEsch]

Why should package-lock.json be excluded from Git? Node.js documentation states the opposite: https://nodejs.dev/learn/the-package-lock-json-file

Kind regards,

Sebastian

[SebastianEsch]

Same here: https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json

[MarcusSchölzel]

I agree with Gregor and the others that the best practice is to commit the package-lock and use "npm ci" to get a reproducable installation (btw not consistently used by SAP generators and buildpacks). I think it is a good option for beginners or BAS only users, but it should be possible to override this artifact repository somehow (with .npmrc it is currently not possible).

But it looks like they are currently doing it only for npm. Yarn is accessing the public repo directly. And i think that yarn is also installed in the default CF buildpack.

So my solution, for the time being, would be to switch to yarn.

[former_member6792]

Hi Marcus, I agree with you (with .npmrc ) it is not overriding. Even with manual overriding in package-lock.json it is having the same error on cf push. Still there is no solution from SAP End

[camichel]

After I used all your hints...i still got some entries in the package-lock with the nginx-redirector which leads in the deploy process to an error 😞 any idea? Started the whole process in BAS?!

[SebastianEsch]

I would either remove the offending entries from package-lock.json or delete package-lock.json completely and then run npm install again. Make sure if you do this in BAS that you have an .npmrc file in place as described above.

[former_member512713]

It seems that the logic in npm CLI for deciding on those registry entries is none trivial, so additional steps are needed to re-generated the package-lock.json with the correct registries entries.

• See: https://github.com/npm/npm/issues/19578#issuecomment-386428859

Note that performing all these "cleanup" steps may result in different versions of the dependencies in the lock file.
So a find/replace may be the safest / most consistent solution.

[i501132]

Thank you Shachar! I followed the steps and now is working for me:

create .npmrc with URL: registry=https://registry.npmjs.com/

Delete package-lock.json

Run npm cache clean -f

Run npm i

cds push