Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use relative keynames in storage and export files #51

Closed
6 tasks done
iandonnelly opened this issue Aug 12, 2014 · 16 comments
Closed
6 tasks done

use relative keynames in storage and export files #51

iandonnelly opened this issue Aug 12, 2014 · 16 comments
Labels
Milestone

Comments

@iandonnelly
Copy link
Contributor

iandonnelly commented Aug 12, 2014

  • simpleini
  • ni
  • tcl
  • dump
  • yajl with cascading keys
  • xmltool (not really relevant, is superseded by xerces plugin)

I am currently having a problem with kdb import. When I try to import a file:
cat backup.ecf | sudo kdb import system/restore
None of the keys are added from backup.ecf (which is in dump format).

If I have a file such as backup.txt in line format,
cat backup.txt | sudo kdb import system/restore line
Works as expected.

Can anybody confirm?

@markus2330
Copy link
Contributor

Without the content of backup.ecf it could be any issue, but I think you stumbled over the fact that dump only uses absolute names and that kdb import cannot rename keys.

See export_dump.md (please also write about that in the tutorials):

$ kdb set system/export_test/a b
create a new key system/export_test/a with string b

$ kdb export system/export_test > e.ecf

$ cat e.ecf
kdbOpen 1
ksNew 1
keyNew 21 2
system/export_test/ab
keyEnd
ksEnd

$ kdb rm system/export_test/a

$ kdb get system/export_test/a
Did not find key

$ cat e.ecf | kdb import system/export_test

$ kdb get system/export_test/a
b

$ cat e.ecf | kdb import system/export_test_renamed

$ kdb get system/export_test_renamed/a
Did not find key

$ kdb get system/export_test/a
b

Does this answer the question or did you stumble over another issue?

@iandonnelly
Copy link
Contributor Author

That was it, I thought I specifically checked that but apparently not well enough, working now. Other formats don't have this limitation, correct?

@iandonnelly
Copy link
Contributor Author

I want to add that I think that we should handle this better though. Rather than not importing at all, we should also print a message to the console informing the user of the error.

@markus2330
Copy link
Contributor

There are some plugins that use absolute names, it should be mentioned in the contract, I added a TODO in CONTRACT.ini to add an explicit field for that information.

It is not really an error, because the user requested that just keys below system/export_test_renamed should be imported (but there are none).

Its like doing

  kdb ls system/export_test_renamed

and warning about not getting keys below system/export_test

We could, however, print information about which keys are going to be dropped in verbose mode.

Currently, they are not dropped, and kdb get system/export_test/a will be set, so lets keep the issue open and let us work out these situations.

@markus2330 markus2330 changed the title kdb import not working with dump format dump uses absolute keynames Aug 14, 2014
@markus2330
Copy link
Contributor

@easyxtarget you are right that it is not very obvious that dump has different behaviour than the others. The advantage dump has that it can contain user + system keys in one export file. (Currently, however, is this not possible to create such files because export does not have the feature)

So this absolute/relative stuff is certainly worth some considerations, but not for this release. (We need a backwards compatible solution anyway)

@markus2330 markus2330 added this to the 0.8.9 milestone Sep 3, 2014
markus2330 pushed a commit that referenced this issue Sep 29, 2014
@markus2330 markus2330 modified the milestones: 0.8.10, 0.8.9 Nov 6, 2014
@markus2330
Copy link
Contributor

further plugins that use absolute names: ni, tcl, xml

@markus2330 markus2330 removed this from the 0.8.10 milestone Dec 3, 2014
@markus2330 markus2330 added bug and removed question labels Dec 3, 2014
@markus2330
Copy link
Contributor

ni is fixed in b91ba11

so we still have (at least): tcl, xmltool and dump

@sanssecours sanssecours self-assigned this Feb 18, 2016
@markus2330
Copy link
Contributor

Some notes: doc/decisions/relative.md describes a bit more the background.

And as scattert in some todos (e.g. TOOLS) the idea is that kdb export does an export for each namespace, making development of plugins easier.

E.g. when you do kdb export /hello ni you would get following output:

kdb import user/hello ni << UNIQUE_EOF
relative/path/below/user/hello
...
UNIQUE_EOF
kdb import system/hello ni << UNIQUE_EOF
relative/path/below/system/hello
...
UNIQUE_EOF

So one can simply execute the exported file and gets everything reimported without having to care about which parts are exported with which format.
(we

As said for dump we need some compatibility so that old dump files (with absolute pathes) can still be read and even be written if not configured to use the new-style.

@markus2330 markus2330 changed the title dump uses absolute keynames use relative keynames in storage and export files Feb 19, 2016
@markus2330
Copy link
Contributor

@sanssecours do you have something ready for this PR? E.g. the fix for cascading export/import?

@sanssecours
Copy link
Member

@sanssecours do you have something ready for this PR? E.g. the fix for cascading export/import?

No. Otherwise I would have opened a pull request.

@markus2330
Copy link
Contributor

Ok, I actually meant "which could be ready for this release" (which is unlikely to happen today or tomorrow, too many issues are still open). Maybe we should at least output a warning that cascading import/export currently is broken for non-relative plugins? Or even tag plugins if they are relative or absolute and have different import/export code. (I would prefer to avoid it, but it seems unlikely that all plugins will be fixed anytime soon, or will they?)

@sanssecours
Copy link
Member

Ok, I actually meant "which could be ready for this release" (which is unlikely to happen today or tomorrow, too many issues are still open).

Sorry, I do not think so. Unless the release is in a few weeks/months.

Maybe we should at least output a warning that cascading import/export currently is broken for non-relative plugins? Or even tag plugins if they are relative or absolute and have different import/export code. (I would prefer to avoid it, but it seems unlikely that all plugins will be fixed anytime soon, or will they?)

I think tagging the sounds like a good idea. Although, ultimately I think we should just fix this issue. You are right though: I assume it will take quite some time until I am done with this.

@markus2330
Copy link
Contributor

Ok, thanks for the update. You might over-estimate it: Once you have done one of the plugins, the others are really similar.

@markus2330
Copy link
Contributor

This is also a serious issue for the web service, where the real path does not match what the user believes the absolute path is.

@sanssecours sanssecours added this to the 1.0.0 milestone Oct 15, 2017
@markus2330 markus2330 modified the milestones: 1.0.0, 0.9.0 Mar 15, 2018
@markus2330 markus2330 mentioned this issue Mar 21, 2018
34 tasks
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 8, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 8, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 9, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 9, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 9, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 9, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 9, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 10, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 11, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 11, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 12, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 12, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 13, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 16, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 16, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 17, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 18, 2018
This update addresses part of issue ElektraInitiative#51.
sanssecours added a commit to sanssecours/elektra that referenced this issue Jun 18, 2018
This update addresses part of issue ElektraInitiative#51.
@markus2330
Copy link
Contributor

I close this issue as dump and xmltool are soon only relevant as legacy formats. (See #2330 #1327)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants