[Rock-dev] clang vs. typelib

[Martin Zenzes]

Hi,

playing around a little bit, reading into this nice codebase of
LLVM/clang and talking to Matthias. Concerned with this horrible state
of gccxml -- no offense -- but it does not make sense to force it to
cross-compile if it is going to be replaced sooner than later. So:

What does a replacement for gccxml in typelib have to provide:
- the full qualifying names of all types defined in a given set of
header-files
- their memory layout: offset+type of all members, the sizeof the type
itself
right?

Nice would be:
- bail out if types use too fancy features, like virtual functions or
dynamic members (except some whitelisted special cases)

The proof-of-concept-level stuff I have so far is living in
$WIESE/temp/mzenzes/clang-typelib.bare but it would make more sense to
push it out into the internet -- should i use my private github account
for things like this?

--


The second point can be done using "clang -fdump-record-layouts" (see
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2014-July/037778.html. It
results in an output like this:

0 | struct base::JointState
0 | double position
8 | float speed
12 | float effort
16 | float raw
| [sizeof=24, dsize=20, align=8
| nvsize=20, nvalign=8]

for the first point (and third) "clang-check -ast-list" could be used --
but it prints too much. So a small clang-tool with added filtering might
come in handy?

Greetings
Martin
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Martin Zenzes]

Hi,

playing around a little bit, reading into this nice codebase of
LLVM/clang and talking to Matthias. Concerned with this horrible state
of gccxml -- no offense -- but it does not make sense to force it to
cross-compile if it is going to be replaced sooner than later. So:

What does a replacement for gccxml in typelib have to provide:
- the full qualifying names of all types defined in a given set of
header-files
- their memory layout: offset+type of all members, the sizeof the type
itself
right?

Nice would be:
- bail out if types use too fancy features, like virtual functions or
dynamic members (except some whitelisted special cases)

The proof-of-concept-level stuff I have so far is living in
$WIESE/temp/mzenzes/clang-typelib.bare but it would make more sense to
push it out into the internet -- should i use my private github account
for things like this?

--


The second point can be done using "clang -fdump-record-layouts" (see
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2014-July/037778.html. It
results in an output like this:

0 | struct base::JointState
0 | double position
8 | float speed
12 | float effort
16 | float raw
| [sizeof=24, dsize=20, align=8
| nvsize=20, nvalign=8]

for the first point (and third) "clang-check -ast-list" could be used --
but it prints too much. So a small clang-tool with added filtering might
come in handy?

Greetings
Martin

ps: sorry if this is posted twice... my mail client?
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Janosch Machowinski]

Post by Martin Zenzes
Hi,
playing around a little bit, reading into this nice codebase of
LLVM/clang and talking to Matthias. Concerned with this horrible state
of gccxml -- no offense -- but it does not make sense to force it to
- the full qualifying names of all types defined in a given set of
header-files
- their memory layout: offset+type of all members, the sizeof the type
itself
right?

Jup, plus name of the member itself ;-P

Post by Martin Zenzes
- bail out if types use too fancy features, like virtual functions or
dynamic members (except some whitelisted special cases)

Actually this is needed, as typlelib can't deal with any class, that has a
virtual table.

Post by Martin Zenzes
The proof-of-concept-level stuff I have so far is living in
$WIESE/temp/mzenzes/clang-typelib.bare but it would make more sense to
push it out into the internet -- should i use my private github account
for things like this?

Sound like a good idea.

Post by Martin Zenzes
--
The second point can be done using "clang -fdump-record-layouts" (see
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2014-July/037778.html. It
0 | struct base::JointState
0 | double position
8 | float speed
12 | float effort
16 | float raw
| [sizeof=24, dsize=20, align=8
| nvsize=20, nvalign=8]
for the first point (and third) "clang-check -ast-list" could be used --
but it prints too much. So a small clang-tool with added filtering might
come in handy?
Greetings
Martin
ps: sorry if this is posted twice... my mail client?

--
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik

Universit?t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra?e 1
28359 Bremen, Germany

Zentrale: [phone]

Besuchsadresse der Nebengesch?ftstelle:
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Tel.: [phone]
Empfang: [phone]
Fax: [phone]
E-Mail: jmachowinski at informatik.uni-bremen.de

Weitere Informationen: http://www.informatik.uni-bremen.de/robotik

[Martin Zenzes]

clarifications...

Post by Janosch Machowinski

Post by Martin Zenzes
- the full qualifying names of all types defined in a given set of
header-files
- their memory layout: offset+type of all members, the sizeof the type
itself
right?

Jup, plus name of the member itself ;-P

what i meant: get a list of all types defined in a a given header file--
so that only the types defined in a certain set of headers are seen by
typelib at all. prior to obtaining the actual memory-layout. this would
save us a lot of processing, and keep the tlb-files smaller

I pushed the stuff i have to far to
https://github.com/marvin2k/clang-typelib.git
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Post by Martin Zenzes
what i meant: get a list of all types defined in a a given header file--
so that only the types defined in a certain set of headers are seen by
typelib at all. prior to obtaining the actual memory-layout. this would
save us a lot of processing, and keep the tlb-files smaller

That's exactly what we need. Typelib currently has to filter it out of
the gccxml output (which makes the gccxml output horribly big).

Thanks for getting into it, we all agree that getting rid of gccxml is
a very very nice to have ...

Sylvain

[Sylvain Joyeux]

One question: why do you have to provide the 'build' directory to the tool ?

Post by Sylvain Joyeux

Post by Martin Zenzes
what i meant: get a list of all types defined in a a given header file--
so that only the types defined in a certain set of headers are seen by
typelib at all. prior to obtaining the actual memory-layout. this would
save us a lot of processing, and keep the tlb-files smaller

That's exactly what we need. Typelib currently has to filter it out of
the gccxml output (which makes the gccxml output horribly big).
Thanks for getting into it, we all agree that getting rid of gccxml is
a very very nice to have ...
Sylvain

[Sylvain Joyeux]

One good news: it appears that we *can* buil the tool out-of-tree ...
Having to get llvm / clang each time is really not a good way forward.
I am fighting against the "right" list of libraries that are needed to
get it right ...

Post by Sylvain Joyeux
One question: why do you have to provide the 'build' directory to the tool ?

Post by Sylvain Joyeux

Post by Martin Zenzes
what i meant: get a list of all types defined in a a given header file--
so that only the types defined in a certain set of headers are seen by
typelib at all. prior to obtaining the actual memory-layout. this would
save us a lot of processing, and keep the tlb-files smaller

That's exactly what we need. Typelib currently has to filter it out of
the gccxml output (which makes the gccxml output horribly big).
Thanks for getting into it, we all agree that getting rid of gccxml is
a very very nice to have ...
Sylvain

[Matthias Goldhoorn]

Morning,
This was the point where i'm giving up.
The linking process is the hell, the llvm-config-3.4 --cxxflags does not
help either.

Theoretical it sould be possible.
Unforntalty i don't have much time within the next weeks to continue my
work on the clang stuff, but i updated martin with my current last state..


Best,
Matthias

Post by Sylvain Joyeux
Having to get llvm / clang each time is really not a good way forward.
I am fighting against the "right" list of libraries that are needed to
get it right ...

Post by Sylvain Joyeux
One question: why do you have to provide the 'build' directory to the tool ?

Post by Sylvain Joyeux

Post by Martin Zenzes
what i meant: get a list of all types defined in a a given header file--
so that only the types defined in a certain set of headers are seen by
typelib at all. prior to obtaining the actual memory-layout. this would
save us a lot of processing, and keep the tlb-files smaller

That's exactly what we need. Typelib currently has to filter it out of
the gccxml output (which makes the gccxml output horribly big).
Thanks for getting into it, we all agree that getting rid of gccxml is
a very very nice to have ...
Sylvain

_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
Dipl.-Inf. Matthias Goldhoorn
Space and Underwater Robotic

Universit?t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra?e 1
28359 Bremen, Germany

Zentrale: [phone]

Besuchsadresse der Nebengesch?ftstelle:
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Tel.: [phone]
Empfang: [phone]
Fax: [phone]
E-Mail: matthias.goldhoorn at informatik.uni-bremen.de

Weitere Informationen: http://www.informatik.uni-bremen.de/robotik

[Sylvain Joyeux]

On Wed, Jul 16, 2014 at 10:29 AM, Matthias Goldhoorn

Post by Matthias Goldhoorn
Morning,
This was the point where i'm giving up.
The linking process is the hell, the llvm-config-3.4 --cxxflags does not

Actually, llvm-config-3.4 helps a lot. The issue is that there are no
equivalent tool for clang.

Sylvai

[Martin Zenzes]

Post by Sylvain Joyeux
Having to get llvm / clang each time is really not a good way forward.
I am fighting against the "right" list of libraries that are needed to
get it right ...

well building inside the clang source tree is the easiest way, while in
the proof-of-concept phase.

ubuntu 12.04 comes with libclang-3.0 (hmm...)
ubuntu 14.04 comes with libclang-3.4
debian comes with 3.4 and 3.3...

so as long as we only link and don't patch i don't see a reason to
compile libclang at all? maybe 3.0 is a bit too old, but i can't see in
that direction yet. even than, tools like "YouCompleteMe" (syntax
completion engine for vim) downloads a libclang-3.4 tar.gz from their
servers and links to it. no compilation.

and getting the right set of libs after the proof-of-concept phase could
be hardcoded in the typelib-cmakelists?

Post by Sylvain Joyeux

Post by Sylvain Joyeux
One question: why do you have to provide the 'build' directory to the tool ?

Post by Sylvain Joyeux

Post by Martin Zenzes
what i meant: get a list of all types defined in a a given header file--
so that only the types defined in a certain set of headers are seen by
typelib at all. prior to obtaining the actual memory-layout. this would
save us a lot of processing, and keep the tlb-files smaller

That's exactly what we need. Typelib currently has to filter it out of
the gccxml output (which makes the gccxml output horribly big).
Thanks for getting into it, we all agree that getting rid of gccxml is
a very very nice to have ...
Sylvain

--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Post by Martin Zenzes
well building inside the clang source tree is the easiest way, while in
the proof-of-concept phase.

Oh ... definitely

Post by Martin Zenzes
ubuntu 12.04 comes with libclang-3.0 (hmm...)
ubuntu 14.04 comes with libclang-3.4
debian comes with 3.4 and 3.3...

I would make people stick with gccxml on 12.04 and support clang only
from 14.04 onwards. By the time we get it to work properly, we will be
close to stop the 12.04 support anyways.

Sylvain

[Sylvain Joyeux]

Something else that can be of interest:

https://github.com/tritao/CppSharp/blob/master/src/Parser/Parser.cpp

it is C#, but does really what we need (and is really really close to
C++ anyways).

Sylvain

Post by Sylvain Joyeux

Post by Martin Zenzes
well building inside the clang source tree is the easiest way, while in
the proof-of-concept phase.

Oh ... definitely

Post by Martin Zenzes
ubuntu 12.04 comes with libclang-3.0 (hmm...)
ubuntu 14.04 comes with libclang-3.4
debian comes with 3.4 and 3.3...

I would make people stick with gccxml on 12.04 and support clang only
from 14.04 onwards. By the time we get it to work properly, we will be
close to stop the 12.04 support anyways.
Sylvain

[Martin Zenzes]

Post by Sylvain Joyeux
https://github.com/tritao/CppSharp/blob/master/src/Parser/Parser.cpp
it is C#, but does really what we need (and is really really close to
C++ anyways).

uhaa, 2200 lines? i have the hope that a single (complicated) line of
AST-matcher plus some if-filter in the callback (for the list of types
in a header file) and a call to the compiler-backend (for the memory
layout) is sufficient...?

Post by Sylvain Joyeux
Sylvain

Post by Sylvain Joyeux

Post by Martin Zenzes
well building inside the clang source tree is the easiest way, while in
the proof-of-concept phase.

Oh ... definitely

Post by Martin Zenzes
ubuntu 12.04 comes with libclang-3.0 (hmm...)
ubuntu 14.04 comes with libclang-3.4
debian comes with 3.4 and 3.3...

I would make people stick with gccxml on 12.04 and support clang only
from 14.04 onwards. By the time we get it to work properly, we will be
close to stop the 12.04 support anyways.
Sylvain

--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Well.. They are wrapping complete libraries, so they need a lot more.

I was giving it out as an example of:
- how to get the binary layout / set up the parser to get it (around line 586)
- how to set up the stuff without a compilation database (they give
arguments directly)

Now, if you feel that you don't need it, feel free to ignore ;-)

Sylvain

Post by Martin Zenzes

Post by Sylvain Joyeux
https://github.com/tritao/CppSharp/blob/master/src/Parser/Parser.cpp
it is C#, but does really what we need (and is really really close to
C++ anyways).

uhaa, 2200 lines? i have the hope that a single (complicated) line of
AST-matcher plus some if-filter in the callback (for the list of types
in a header file) and a call to the compiler-backend (for the memory
layout) is sufficient...?

Post by Sylvain Joyeux
Sylvain

Post by Sylvain Joyeux

Post by Martin Zenzes
well building inside the clang source tree is the easiest way, while in
the proof-of-concept phase.

Oh ... definitely

Post by Martin Zenzes
ubuntu 12.04 comes with libclang-3.0 (hmm...)
ubuntu 14.04 comes with libclang-3.4
debian comes with 3.4 and 3.3...

I would make people stick with gccxml on 12.04 and support clang only
from 14.04 onwards. By the time we get it to work properly, we will be
close to stop the 12.04 support anyways.
Sylvain

--
M.Sc. Martin Zenzes
Space Robotics
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany
Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

[Martin Zenzes]

Post by Sylvain Joyeux
Well.. They are wrapping complete libraries, so they need a lot more.
- how to get the binary layout / set up the parser to get it (around line 586)

ahjo, janosch?

Post by Sylvain Joyeux
- how to set up the stuff without a compilation database (they give
arguments directly)

yes, this is possible of course. there is no need to set up stuff
manually -- as far is I got reading the documentation the
"ParseCommandLineOptions" can handle this.

Post by Sylvain Joyeux
Now, if you feel that you don't need it, feel free to ignore ;-)

okok... I just saw 2000 lines of code... ;-)

Post by Sylvain Joyeux
Sylvain

Post by Martin Zenzes

Post by Sylvain Joyeux
https://github.com/tritao/CppSharp/blob/master/src/Parser/Parser.cpp
it is C#, but does really what we need (and is really really close to
C++ anyways).

uhaa, 2200 lines? i have the hope that a single (complicated) line of
AST-matcher plus some if-filter in the callback (for the list of types
in a header file) and a call to the compiler-backend (for the memory
layout) is sufficient...?

Post by Sylvain Joyeux
Sylvain

Post by Sylvain Joyeux

Post by Martin Zenzes
well building inside the clang source tree is the easiest way, while in
the proof-of-concept phase.

Oh ... definitely

Post by Martin Zenzes
ubuntu 12.04 comes with libclang-3.0 (hmm...)
ubuntu 14.04 comes with libclang-3.4
debian comes with 3.4 and 3.3...

I would make people stick with gccxml on 12.04 and support clang only
from 14.04 onwards. By the time we get it to work properly, we will be
close to stop the 12.04 support anyways.
Sylvain

--
M.Sc. Martin Zenzes
Space Robotics
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany
Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Martin Zenzes]

Post by Sylvain Joyeux
One question: why do you have to provide the 'build' directory to the tool ?

this is just how the tool-template from the clang/tools/extra wants it
do do. i think it looks up the compile_commands database to get the
needed flags for a compile unit...
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Post by Martin Zenzes

Post by Sylvain Joyeux
One question: why do you have to provide the 'build' directory to the tool ?

this is just how the tool-template from the clang/tools/extra wants it
do do. i think it looks up the compile_commands database to get the
needed flags for a compile unit...

OK ... Let's stick with that for now, we'll figure it out later

Sylvain

[Martin Zenzes]

I don't know where to continue in this thread ;-)

so, after another day we are at this stage:

./bin/extractor $ROCK_ROOT/base/types/base/Pose.hpp --
-I$ROCK_ROOT/base/types -I/usr/include/eigen3 -x c++
... (output see below)
15.47s user 0.05s system 99% cpu 15.519 total

which is the only downpoint at this stage -- 15seconds for one header...
not sure where the time is spent, leaving out the ast_matcher (basically
running an empty finder which does nothing) does not change the amount
of time at all... well, this is for tomorrow...

only the declarations in the given header file are outputted. no json
database is needed, the options after the double-dash are normal options
passed to the compiler.

greetings
martin

got 'struct' named 'base::PoseUpdateThreshold' in
'/home/mzenzes/TransTerrA/rock/base/types/base/Pose.hpp:74:12'
Type base::PoseUpdateThreshold
Size 16
Members :
Field base::PoseUpdateThreshold::distance
Offset 0
Decl Name distance
Qual Type double

Field base::PoseUpdateThreshold::angle
Offset 64
Decl Name angle
Qual Type double

got 'struct' named 'base::Pose' in
'/home/mzenzes/TransTerrA/rock/base/types/base/Pose.hpp:127:12'
Type base::Pose
Size 56
Members :
Field base::Pose::position
Offset 0
Decl Name position
Qual Type Position

Field base::Pose::orientation
Offset 192
Decl Name orientation
Qual Type Orientation

got 'struct' named 'base::Pose2D' in
'/home/mzenzes/TransTerrA/rock/base/types/base/Pose.hpp:228:12'
Type base::Pose2D
Size 24
Members :
Field base::Pose2D::position
Offset 0
Decl Name position
Qual Type Position2D

Field base::Pose2D::orientation
Offset 128
Decl Name orientation
Qual Type Orientation2D
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Just tried it out ... clang generates piles of errors because it is
trying to include GCC headers. Did you encounter anything like it ?

One thing you may have missed: types need to be resolved recursively.
That is, you need to resolve the fields types, and the fields types of
the fields types and so on and so forth.

Other than that, it really looks promising ...

Sylvain

Post by Martin Zenzes
I don't know where to continue in this thread ;-)
./bin/extractor $ROCK_ROOT/base/types/base/Pose.hpp --
-I$ROCK_ROOT/base/types -I/usr/include/eigen3 -x c++
... (output see below)
15.47s user 0.05s system 99% cpu 15.519 total
which is the only downpoint at this stage -- 15seconds for one header...
not sure where the time is spent, leaving out the ast_matcher (basically
running an empty finder which does nothing) does not change the amount
of time at all... well, this is for tomorrow...
only the declarations in the given header file are outputted. no json
database is needed, the options after the double-dash are normal options
passed to the compiler.
greetings
martin
got 'struct' named 'base::PoseUpdateThreshold' in
'/home/mzenzes/TransTerrA/rock/base/types/base/Pose.hpp:74:12'
Type base::PoseUpdateThreshold
Size 16
Field base::PoseUpdateThreshold::distance
Offset 0
Decl Name distance
Qual Type double
Field base::PoseUpdateThreshold::angle
Offset 64
Decl Name angle
Qual Type double
got 'struct' named 'base::Pose' in
'/home/mzenzes/TransTerrA/rock/base/types/base/Pose.hpp:127:12'
Type base::Pose
Size 56
Field base::Pose::position
Offset 0
Decl Name position
Qual Type Position
Field base::Pose::orientation
Offset 192
Decl Name orientation
Qual Type Orientation
got 'struct' named 'base::Pose2D' in
'/home/mzenzes/TransTerrA/rock/base/types/base/Pose.hpp:228:12'
Type base::Pose2D
Size 24
Field base::Pose2D::position
Offset 0
Decl Name position
Qual Type Position2D
Field base::Pose2D::orientation
Offset 128
Decl Name orientation
Qual Type Orientation2D
--
M.Sc. Martin Zenzes
Space Robotics
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany
Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

[Martin Zenzes]

Post by Sylvain Joyeux
Just tried it out ... clang generates piles of errors because it is
trying to include GCC headers. Did you encounter anything like it ?

rhetorical standard question: which os? which llvm-branch?
did u "make all" in the llvm-build dir, or just "make extractor"?

I got this kind of error "often" when using clang-based tooling. there
is the "-isystem" option, which is able to fix this (for example this
[1]). but this this feels kinda hacky ;-) for me the "release_34" branch
builds, links and runs just fine on Debian testing

Post by Sylvain Joyeux
One thing you may have missed: types need to be resolved recursively.
That is, you need to resolve the fields types, and the fields types of
the fields types and so on and so forth.

we talked about this, and said that these types are pulled later from
the typelib-registry? no need to resolve them recursively at the level
of the clang-based-tool? (which still needs a proper name...)

Post by Sylvain Joyeux
Other than that, it really looks promising ...

I'm suprised how fast we got to this stage... What would be the next steps?
- definitely speedup
- figuring out how to link against a pre-packaged clang installed in the
system
- test with clang-3.0

[1] https://github.com/Valloric/YouCompleteMe/issues/303

[Sylvain Joyeux]

rhetorical standard question lwhich os? which llvm-branch?
did u "make all" in the llvm-build dir, or just "make extractor"?

Mmmmm... One important "detail". It is an out-of-build tree (I got it
to work), so prebuilt clang 3.4 from Ubuntu.

Post by Sylvain Joyeux
One thing you may have missed: types need to be resolved recursively.
That is, you need to resolve the fields types, and the fields types of
the fields types and so on and so forth.

we talked about this, and said that these types are pulled later from
the typelib-registry? no need to resolve them recursively at the level
of the clang-based-tool? (which still needs a proper name...)

Eeeeh. Misunderstanding.
Nope, they cannot be pulled later from the typelib registry as that
would require the user to basically tell orogen all the files that
should be pulled (replicating the set of recursive includes). If you
think about it, really not fun when you start hitting Eigen or boost.

Sylvain

[Sylvain Joyeux]

The out-of-tree build is there, by the way

https://github.com/doudou/clang-typelib/tree/out_of_tree

Post by Sylvain Joyeux

rhetorical standard question lwhich os? which llvm-branch?
did u "make all" in the llvm-build dir, or just "make extractor"?

Mmmmm... One important "detail". It is an out-of-build tree (I got it
to work), so prebuilt clang 3.4 from Ubuntu.

Post by Sylvain Joyeux
One thing you may have missed: types need to be resolved recursively.
That is, you need to resolve the fields types, and the fields types of
the fields types and so on and so forth.

we talked about this, and said that these types are pulled later from
the typelib-registry? no need to resolve them recursively at the level
of the clang-based-tool? (which still needs a proper name...)

Eeeeh. Misunderstanding.
Nope, they cannot be pulled later from the typelib registry as that
would require the user to basically tell orogen all the files that
should be pulled (replicating the set of recursive includes). If you
think about it, really not fun when you start hitting Eigen or boost.
Sylvain

[Sylvain Joyeux]

Another thing that we did not mention: typedefs.

Typedefs need to be registered as well. It is really critical for
complex templates that the user can use base/Vector3d instead of the
complete eigen definition ...

Sylvain

Post by Sylvain Joyeux
The out-of-tree build is there, by the way
https://github.com/doudou/clang-typelib/tree/out_of_tree

Post by Sylvain Joyeux

rhetorical standard question lwhich os? which llvm-branch?
did u "make all" in the llvm-build dir, or just "make extractor"?

Mmmmm... One important "detail". It is an out-of-build tree (I got it
to work), so prebuilt clang 3.4 from Ubuntu.

Post by Sylvain Joyeux
One thing you may have missed: types need to be resolved recursively.
That is, you need to resolve the fields types, and the fields types of
the fields types and so on and so forth.

we talked about this, and said that these types are pulled later from
the typelib-registry? no need to resolve them recursively at the level
of the clang-based-tool? (which still needs a proper name...)

Eeeeh. Misunderstanding.
Nope, they cannot be pulled later from the typelib registry as that
would require the user to basically tell orogen all the files that
should be pulled (replicating the set of recursive includes). If you
think about it, really not fun when you start hitting Eigen or boost.
Sylvain

[Janosch Machowinski]

So,
basically I would register a alias, per 'DeSugaring' step I do ?!
DeSugaring is the name in clang, for removing one layer
of typedefs..

Post by Sylvain Joyeux
Another thing that we did not mention: typedefs.
Typedefs need to be registered as well. It is really critical for
complex templates that the user can use base/Vector3d instead of the
complete eigen definition ...
Sylvain

Post by Sylvain Joyeux
Eeeeh. Misunderstanding.
Nope, they cannot be pulled later from the typelib registry as that
would require the user to basically tell orogen all the files that
should be pulled (replicating the set of recursive includes). If you
think about it, really not fun when you start hitting Eigen or boost.

We could also place references to TLB files in TLB files themself,
and generate a TLB File per header file. In this case we could stop
parsing anything, it there is already a TLB file for an included
header. This could lead to big speedups, as we don't reparse all the
headers over and over again.
Greetings
Janosch
--
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik

Universit?t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra?e 1
28359 Bremen, Germany

Zentrale: [phone]

Besuchsadresse der Nebengesch?ftstelle:
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Tel.: [phone]
Empfang: [phone]
Fax: [phone]
E-Mail: jmachowinski at informatik.uni-bremen.de

Weitere Informationen: http://www.informatik.uni-bremen.de/robotik

[Sylvain Joyeux]

Post by Janosch Machowinski
We could also place references to TLB files in TLB files themself,
and generate a TLB File per header file. In this case we could stop
parsing anything, it there is already a TLB file for an included
header. This could lead to big speedups, as we don't reparse all the
headers over and over again.

Before going into modifications on the typelib side, I would rather
look into why it is so slow (and ask the clang mailing list how to
make it faster). Compiling a file that includes base/Pose.hpp and does
something with a base::Pose takes 1.61s on my machine, not 15s. Maybe
there would be a way to prune the traversal at the declaration level
since we really don't care about the inside of the methods.

If we come to that, I would really really not break the self-contained
nature of tlb files, because it won't be manageable (if you update a
linked tlb file because a struct change but NOT the other ones, you
are breaking the type definitions). What would definitely be doable is
to save a self-contained TLB per header file in a cache folder and
have the clang importer merge these tlbs into the current registry
whenever an include is found. Now, how do you detect the includes
without traversing the AST ? And -- more importantly -- how do you
make sure that the cache is properly invalidated when some
include-in-include-in-include file gets invalidated ? Or when the
import options (such as the opaque list) gets modified ? Big can of
worms.

Sylvain

[Janosch Machowinski]

Post by Sylvain Joyeux

Post by Janosch Machowinski
We could also place references to TLB files in TLB files themself,
and generate a TLB File per header file. In this case we could stop
parsing anything, it there is already a TLB file for an included
header. This could lead to big speedups, as we don't reparse all the
headers over and over again.

Before going into modifications on the typelib side, I would rather
look into why it is so slow (and ask the clang mailing list how to
make it faster). Compiling a file that includes base/Pose.hpp and does
something with a base::Pose takes 1.61s on my machine, not 15s. Maybe
there would be a way to prune the traversal at the declaration level
since we really don't care about the inside of the methods.

Definitely.
This comment was more about the logic, how to convert the AST
to Typelib later on. How would the final API to this tool look like ?
- Give Types, and Headers, to put into a TLB ?
- Dump every Type from a set of Headers into a TLB ?

Janosch

Post by Sylvain Joyeux
If we come to that, I would really really not break the self-contained
nature of tlb files, because it won't be manageable (if you update a
linked tlb file because a struct change but NOT the other ones, you
are breaking the type definitions). What would definitely be doable is
to save a self-contained TLB per header file in a cache folder and
have the clang importer merge these tlbs into the current registry
whenever an include is found. Now, how do you detect the includes
without traversing the AST ? And -- more importantly -- how do you
make sure that the cache is properly invalidated when some
include-in-include-in-include file gets invalidated ? Or when the
import options (such as the opaque list) gets modified ? Big can of
worms.
Sylvain
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik

Universit?t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra?e 1
28359 Bremen, Germany

Zentrale: [phone]

Besuchsadresse der Nebengesch?ftstelle:
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Tel.: [phone]
Empfang: [phone]
Fax: [phone]
E-Mail: jmachowinski at informatik.uni-bremen.de

Weitere Informationen: http://www.informatik.uni-bremen.de/robotik

[Sylvain Joyeux]

Well. I personally was seeing it as a typelib plugin (living the
typelib package), which:
- gets a set of headers
- a set of options (mostly -I and -D)
- some information about opaque types
- and outputs a Typelib::Registry object with the (complete)
definition of all the types found in the specified headers.

The tlb importer is an example (in lang/tlb/importer.cc)

Sylvain

On Thu, Jul 17, 2014 at 2:42 PM, Janosch Machowinski

Post by Janosch Machowinski

Post by Sylvain Joyeux

Post by Janosch Machowinski
We could also place references to TLB files in TLB files themself,
and generate a TLB File per header file. In this case we could stop
parsing anything, it there is already a TLB file for an included
header. This could lead to big speedups, as we don't reparse all the
headers over and over again.

Before going into modifications on the typelib side, I would rather
look into why it is so slow (and ask the clang mailing list how to
make it faster). Compiling a file that includes base/Pose.hpp and does
something with a base::Pose takes 1.61s on my machine, not 15s. Maybe
there would be a way to prune the traversal at the declaration level
since we really don't care about the inside of the methods.

Definitely.
This comment was more about the logic, how to convert the AST
to Typelib later on. How would the final API to this tool look like ?
- Give Types, and Headers, to put into a TLB ?
- Dump every Type from a set of Headers into a TLB ?
Janosch

Post by Sylvain Joyeux
If we come to that, I would really really not break the self-contained
nature of tlb files, because it won't be manageable (if you update a
linked tlb file because a struct change but NOT the other ones, you
are breaking the type definitions). What would definitely be doable is
to save a self-contained TLB per header file in a cache folder and
have the clang importer merge these tlbs into the current registry
whenever an include is found. Now, how do you detect the includes
without traversing the AST ? And -- more importantly -- how do you
make sure that the cache is properly invalidated when some
include-in-include-in-include file gets invalidated ? Or when the
import options (such as the opaque list) gets modified ? Big can of
worms.
Sylvain
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik
Universit?t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra?e 1
28359 Bremen, Germany
Zentrale: [phone]
Robert-Hooke-Stra?e 5
28359 Bremen, Germany
Tel.: [phone]
Empfang: [phone]
Fax: [phone]
E-Mail: jmachowinski at informatik.uni-bremen.de
Weitere Informationen: http://www.informatik.uni-bremen.de/robotik
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

[Sylvain Joyeux]

Post by Janosch Machowinski
basically I would register a alias, per 'DeSugaring' step I do ?!
DeSugaring is the name in clang, for removing one layer
of typedefs..

Yes, but starting with the type (when you register an alias, the
aliased type has to exist).

Sylvain

[Martin Zenzes]

Post by Sylvain Joyeux

Post by Janosch Machowinski
basically I would register a alias, per 'DeSugaring' step I do ?!
DeSugaring is the name in clang, for removing one layer
of typedefs..

Yes, but starting with the type (when you register an alias, the
aliased type has to exist).

what would be an example of this?

i added code to recursively desugar nested records in the
extractor-master... certainly not correct or elegant, but its there ;-)

for the example present it looks good, only template argument are not
printed in the moment (eg for Eigen-types)...

Post by Sylvain Joyeux
Sylvain
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Janosch Machowinski]

Okay, I found a way to get all Typedefs.
One needs to match in the AST for their declarations.
Greetings
Janosch

Post by Martin Zenzes

Post by Sylvain Joyeux

Post by Janosch Machowinski
basically I would register a alias, per 'DeSugaring' step I do ?!
DeSugaring is the name in clang, for removing one layer
of typedefs..

Yes, but starting with the type (when you register an alias, the
aliased type has to exist).

what would be an example of this?
i added code to recursively desugar nested records in the
extractor-master... certainly not correct or elegant, but its there ;-)
for the example present it looks good, only template argument are not
printed in the moment (eg for Eigen-types)...

Post by Sylvain Joyeux
Sylvain
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
Dipl. Inf. Janosch Machowinski
SAR- & Sicherheitsrobotik

Universit?t Bremen
FB 3 - Mathematik und Informatik
AG Robotik
Robert-Hooke-Stra?e 1
28359 Bremen, Germany

Zentrale: [phone]

Besuchsadresse der Nebengesch?ftstelle:
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Tel.: [phone]
Empfang: [phone]
Fax: [phone]
E-Mail: jmachowinski at informatik.uni-bremen.de

Weitere Informationen: http://www.informatik.uni-bremen.de/robotik

[Martin Zenzes]

So to make it more clear -- sorry if I'm a bit slow on this, but I like
things black-on-white and crystal-clear:

In the moment (as far as i understand this) _all_ the records and
typedefs from the whole include-chain triggered by a single
"import_from" are stored in a database to be read by orogen. This allows
usage of "struct timespec" although no one explicitly did "import_from
time.h". This also allowed usage of "uint16_t" for properties although
no one explicitly did "import_from stdint.h". This was caused by how
gccxml works. Is this parse-and-store-everything still desired?

Makes the database large -- considering an eigen-based-example with ~200
RecordDecls from system-headers and ~130 from eigen-headers.

Now, given a set of header files, the clang-based tool should:


- give all TypeDecl defined somewhere in the include-chain of the given
header with its CanonicalName and the field-layout.
- If this is a RecordDecl: For each field then do recursively: return
the offset and BuiltinType or the CanonicalName and field-layout


- _all_ mappings "typedef" -> "CanonicalName" declared somewhere in the
include-chain of the given header file.

- give all enum-values as strings


The outputs from different calls are then merged and collected in some
typelib database, however this is organized...

Additionally reject Records with:
- pointers
- virtual functions
- references
- bitfields?
And warn about Records with:
- FieldTypes with architecture-dependent size, like int

Hm?

[Sylvain Joyeux]

Post by Martin Zenzes
In the moment (as far as i understand this) _all_ the records and
typedefs from the whole include-chain triggered by a single
"import_from" are stored in a database to be read by orogen. This allows
usage of "struct timespec" although no one explicitly did "import_from
time.h". This also allowed usage of "uint16_t" for properties although
no one explicitly did "import_from stdint.h". This was caused by how
gccxml works. Is this parse-and-store-everything still desired?

Wrong.

The database contains all the types that are defined in the explicitly
given headers (import_types_from) as well as all the types that are
necessary to define them. If you have a type in your header that uses
timespec, it will end in the database, otherwise it gets thrown out.

Post by Martin Zenzes
- _all_ mappings "typedef" -> "CanonicalName" declared somewhere in the
include-chain of the given header file.

All typedefs pointing to a type that should be included in the database.

Post by Martin Zenzes
- give all enum-values as strings

As string-value pairs.

Post by Martin Zenzes
- pointers
- virtual functions
- references
- bitfields?

Not *exactly*.

Typelib and orogen allows to define "opaque types", which is basically
a way to name a type and tell the system that it is not able to handle
it BUT that it is of interest. In orogen, opaques get converted to a
typelib-compatible type
(http://rock-robotics.org/master/documentation/orogen/opaque_types.html)

During parsing, one MUST define the types that use Records that are
opaques, even if these opaques have pointers, references, virtual
functions, ...

To make it usable for the user, it is allowed to specify an opaque
from a typedef name (i.e. the user says "make base/Vector3d opaque"
instead of Eigen::MatrixBase<double,3,0,0,...>). That also has to be
handled during import.

Finally, the type names are typelib-normalized, not C++-normalized, so
the importer has to handle the convertion. It is currently done in
Ruby (because the gccxml importer is written in Ruby)

Post by Martin Zenzes
- FieldTypes with architecture-dependent size, like int

I would prefer leaving that to as post-processing pass instead of
embedding it in the importer.

Sylvain

[Martin Zenzes]

Post by Sylvain Joyeux
The database contains all the types that are defined in the explicitly
given headers (import_types_from) as well as all the types that are
necessary to define them. If you have a type in your header that uses
timespec, it will end in the database, otherwise it gets thrown out.

Types or Records? I assume Types, so that creating a output-port of
"double" is possible?

Post by Sylvain Joyeux
Not *exactly*.
Typelib and orogen allows to define "opaque types", which is basically
a way to name a type and tell the system that it is not able to handle
it BUT that it is of interest. In orogen, opaques get converted to a
typelib-compatible type
(http://rock-robotics.org/master/documentation/orogen/opaque_types.html)
During parsing, one MUST define the types that use Records that are
opaques, even if these opaques have pointers, references, virtual
functions, ...
To make it usable for the user, it is allowed to specify an opaque
from a typedef name (i.e. the user says "make base/Vector3d opaque"
instead of Eigen::MatrixBase<double,3,0,0,...>). That also has to be
handled during import.

So the importer gets a list of types (typedefs) which are then
whitelisted and the "typedef->CanonicalName" pair is added to the
database. The field-layout of the intermediary type is added by calling
the importer with the wrapper-header.

Post by Sylvain Joyeux

Post by Martin Zenzes
- FieldTypes with architecture-dependent size, like int

I would prefer leaving that to as post-processing pass instead of
embedding it in the importer.

why not warn as early as possible? Sure, in the moment many messages use
"int", so lotta warnings... But the information is here, so... Yeah
well, this is easy to add/leave out... Not that important.
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Post by Martin Zenzes
we talked about this, and said that these types are pulled later from
the typelib-registry? no need to resolve them recursively at the level
of the clang-based-tool? (which still needs a proper name...)

No name: I would personally integrate it as a typelib plugin in the
typelib source tree.

Sylvain

[Sylvain Joyeux]

Next one: running the extractor on base/Pose.hpp my machine (Core
i7-2630QM 2.0 GHz) *with the precompiled clang from ubuntu* takes 1.4s

I had to use -isystem to get it to work.

Sylvain

Post by Sylvain Joyeux

Post by Martin Zenzes
we talked about this, and said that these types are pulled later from
the typelib-registry? no need to resolve them recursively at the level
of the clang-based-tool? (which still needs a proper name...)

No name: I would personally integrate it as a typelib plugin in the
typelib source tree.
Sylvain

[Martin Zenzes]

Post by Sylvain Joyeux
Next one: running the extractor on base/Pose.hpp my machine (Core
i7-2630QM 2.0 GHz) *with the precompiled clang from ubuntu* takes 1.4s

hm... don't ask me, but I was sure to have had enabled a "Release" build
of llvm+clang yesterday... Well, as it goes I had not, did a fresh rebuild:

now it takes 1.5s for "Pose.hpp" on my QuadCore i7 at 2.8GHz... sorry for
the confusion...

Post by Sylvain Joyeux
I had to use -isystem to get it to work.

this look like a ubuntu problem?
--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

No, it is a clang problem.

The default header search path I get when running
./extractor -- -print-search-dirs

is broken: it mentions the clang path as ../lib/clang/3.4, not as an
absolute path. I guess that it has to be given somehow to LLVM/Clang
in the code, but I have no time to investigate further.

Sylvain

Post by Martin Zenzes

Post by Sylvain Joyeux
Next one: running the extractor on base/Pose.hpp my machine (Core
i7-2630QM 2.0 GHz) *with the precompiled clang from ubuntu* takes 1.4s

hm... don't ask me, but I was sure to have had enabled a "Release" build
now it takes 1.5s for "Pose.hpp" on my QuadCore i7 at 2.8GHz... sorry for
the confusion...

Post by Sylvain Joyeux
I had to use -isystem to get it to work.

this look like a ubuntu problem?
--
M.Sc. Martin Zenzes
Space Robotics
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany
Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

[Martin Zenzes]

Post by Sylvain Joyeux
No, it is a clang problem.
The default header search path I get when running
./extractor -- -print-search-dirs
is broken: it mentions the clang path as ../lib/clang/3.4, not as an
absolute path. I guess that it has to be given somehow to LLVM/Clang
in the code, but I have no time to investigate further.

what i meant: this problem is caused by the ubuntu packaging? as far as
i know, these path are hardcoded to llvm/clang at compile time...?

Post by Sylvain Joyeux
Sylvain

Post by Martin Zenzes

Post by Sylvain Joyeux
Next one: running the extractor on base/Pose.hpp my machine (Core
i7-2630QM 2.0 GHz) *with the precompiled clang from ubuntu* takes 1.4s

hm... don't ask me, but I was sure to have had enabled a "Release" build
now it takes 1.5s for "Pose.hpp" on my QuadCore i7 at 2.8GHz... sorry for
the confusion...

Post by Sylvain Joyeux
I had to use -isystem to get it to work.

this look like a ubuntu problem?
--
M.Sc. Martin Zenzes
Space Robotics
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany
Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
M.Sc. Martin Zenzes
Space Robotics

Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: [phone] - 6658
Fax: [phone] - 4150
E-Mail: martin.zenzes at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

[Sylvain Joyeux]

Post by Martin Zenzes
what i meant: this problem is caused by the ubuntu packaging? as far as
i know, these path are hardcoded to llvm/clang at compile time...?

Does not seem to be the case. clang itself is fine. Googling a bit, it
seems that they have a weird handling for search paths.

Sylvain