[Rock-dev] Decision on the fate of external/ ?

Discussion:

Sylvain Joyeux

2014-09-07 19:21:25 UTC

Last try.

We've had the discussion about the desired fate of external/ twice
already. It is time to get a formal vote on it. The current non-policy
makes finding packages seriously annoying. I've been looking for
planning software recently, started thinking that sbpl was not there
only to find that it was indeed there ... in external/

Pro-external
=========
Janosch and Matthias expressed that they feel we need to separate the
"Rock packages" from the "non-Rock packages". That's what external/ is
for. Under this circumstances, the following packages should be moved
to external/
drivers/aravis
drivers/phidgets
drivers/gsf
drivers/aria
control/urdfdom
control/urdfdom_headers
control/reflexxes
data_processing/openann
planning/openrave
control/kdl
image_processing/viso2
image_processing/libelas
slam/pcl
slam/ceres_solver
slam/flann
slam/g2o
slam/mtk
slam/gmapping
slam/octomap
slam/hogman

Con-external/
========================
external/ is a non-category. While we elected to use categories try to
make browsing / finding software easier, external/ by definition
breaks this.

Changing it would require renaming:
external/sbpl => planning/sbpl
external/ompl => planning/ompl
external/freenect => drivers/freenect
external/yaml-cpp => tools/yaml-cpp
external/tinyxml => tools/tinyxml
external/gdal => slam/gdal
external/libply => slam/libply
external/cminpack => ? (math)
external/kdtree => ? (slam/kdtree ?)
external/box2d => ? (2D physics engine but Janosch does not want it
in simulation/)
external/lemon => ? (graphs)
external/snap => ? (graphs)
external/gexf => ? (graphs)
external/aruco => ? (augmented reality)

One additional pro "argument" I've been countered with is that "we
don't know how to categorize package X, so let's put it in external".
When you see the non-categorized packages above, I believe that
finding them a proper category would be a lot more constructive
(because otherwise you mean that e.g. augmented reality or graph
algorithms do not have their place in Rock).

Under the Pro proposal, 1/3 of all the non- packages in the Rock
package set should be in external/. Given that we categorize for the
express purpose of making finding software easier, that sounds like
not such a great situation. Moreover, for Rock to strive, we should
reuse as much as we can (meaning that this proportion should
definitely go up, not down). In the end, don't look for drivers in
drivers/, look for them in drivers/ AND external/. Don't look for
mapping software in slam/, look for slam/ AND external/. You get the
picture.

Finally, I personally feel that if we want to scale up and foster
collaboration, having a policy like this one sounds like "either you
are with us, or you are "external"".

Sylvain

Jakob Schwendner

2014-09-08 07:15:36 UTC

Permalink

Post by Sylvain Joyeux
Last try.
"either you are with us, or you are "external"".

I don't want to be external. I am with you.

Jakob

Sylvain Joyeux

2014-09-08 12:39:37 UTC

Permalink

On Mon, Sep 8, 2014 at 4:15 AM, Jakob Schwendner

Post by Sylvain Joyeux
Last try.
"either you are with us, or you are "external"".

Nice out-of-context quoting ... I actually had to verify the possible
interpretations of what I wrote ...

Sylvain

Matthias Goldhoorn

2014-09-08 08:06:40 UTC

Permalink

Post by Sylvain Joyeux
Last try.
We've had the discussion about the desired fate of external/ twice
already. It is time to get a formal vote on it. The current non-policy
makes finding packages seriously annoying. I've been looking for
planning software recently, started thinking that sbpl was not there
only to find that it was indeed there ... in external/
Pro-external
=========
Janosch and Matthias expressed that they feel we need to separate the
"Rock packages" from the "non-Rock packages". That's what external/ is
for. Under this circumstances, the following packages should be moved
to external/
drivers/aravis
drivers/phidgets
drivers/gsf
drivers/aria
control/urdfdom
control/urdfdom_headers
control/reflexxes
data_processing/openann
planning/openrave
control/kdl
image_processing/viso2
image_processing/libelas
slam/pcl
slam/ceres_solver
slam/flann
slam/g2o
slam/mtk
slam/gmapping
slam/octomap
slam/hogman
Con-external/
========================
external/ is a non-category. While we elected to use categories try to
make browsing / finding software easier, external/ by definition
breaks this.
external/sbpl => planning/sbpl
external/ompl => planning/ompl
external/freenect => drivers/freenect
external/yaml-cpp => tools/yaml-cpp
external/tinyxml => tools/tinyxml
external/gdal => slam/gdal
external/libply => slam/libply
external/cminpack => ? (math)
external/kdtree => ? (slam/kdtree ?)
external/box2d => ? (2D physics engine but Janosch does not want it
in simulation/)
external/lemon => ? (graphs)
external/snap => ? (graphs)
external/gexf => ? (graphs)
external/aruco => ? (augmented reality)
One additional pro "argument" I've been countered with is that "we
don't know how to categorize package X, so let's put it in external".
When you see the non-categorized packages above, I believe that
finding them a proper category would be a lot more constructive
(because otherwise you mean that e.g. augmented reality or graph
algorithms do not have their place in Rock).
Under the Pro proposal, 1/3 of all the non- packages in the Rock
package set should be in external/. Given that we categorize for the
express purpose of making finding software easier, that sounds like
not such a great situation. Moreover, for Rock to strive, we should
reuse as much as we can (meaning that this proportion should
definitely go up, not down). In the end, don't look for drivers in
drivers/, look for them in drivers/ AND external/. Don't look for
mapping software in slam/, look for slam/ AND external/. You get the
picture.
Finally, I personally feel that if we want to scale up and foster
collaboration, having a policy like this one sounds like "either you
are with us, or you are "external"".

In general i don't like the idea of "you are not with us" much, but i
don't like the idea of putting all i "drivers/" too.
We have several packages in drivers/ or external/ which does not belong
there too.
The Problem i have with some packages like "gdal" is that there are
neither a driver nor a "slam/whatever".

Another problem is "aravis" aravis is a library that is needed by the
camera_aravis driver, which is needed by drivers/orogen/aravis.
So we need to have three packages in drivers/ (and drivers/orogen) to
get a camera aravis camera running. Which might also end in some
confusing setups.
Therefore external makes sense for me, we supporting the camera by
integrating it in our camera_aravis package..., otherwise we need to to
the same magic like for the prosilicas which i don't like either.

It's really hard to take a decimation here because as you already
mentions some packages could not really sorted into categories (like
gdal/yaml), putting them in tools/ makes no real difference for me,
pcl/kdtree yould also be only a "tool", kdtree could be used for image
processing too for example.

I thing we should find a way to handle packages that are purly
downloaded and build by rock (maybe patches), and handle them
separately. Therefore was the external folder, but maybe we add
something like dirvers/external slam/external to make the separation
better, this would solve the problem with aravis and a crouwded
external/ meta folder.

For the categorization we should discuss this per package, maybe we can
add symlinks if a package belongs to several categories?

Best,
MAtthias

Post by Sylvain Joyeux
Sylvain
_______________________________________________
Rock-dev mailing list
Rock-dev at dfki.de
http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev

--
--
Matthias Goldhoorn
Unterwasserrobotik

Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: +49 (0)421 218-64100
Fax: +49 (0)421 218-64150
E-Mail: robotik at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

Sylvain Joyeux

2014-09-08 12:37:49 UTC

Permalink

Post by Matthias Goldhoorn
The Problem i have with some packages like "gdal" is that there are
neither a driver nor a "slam/whatever".

Why not ? gdal is a map management / access library. It is as much
relevant to slam/ than envire is.

Post by Matthias Goldhoorn
Another problem is "aravis" aravis is a library that is needed by the
camera_aravis driver, which is needed by drivers/orogen/aravis.

Again, point of view of "in" Rock vs. "out" Rock. From the aravis website:

Aravis is a glib/gobject based library for video acquisition using
Genicam cameras.

This is a driver, with absolutetly zero doubt.

Post by Matthias Goldhoorn
It's really hard to take a decimation here because as you already
mentions some packages could not really sorted into categories (like
gdal/yaml), putting them in tools/ makes no real difference for me,
pcl/kdtree yould also be only a "tool", kdtree could be used for image
processing too for example.

Some packages cannot be put in *existing* categories. I'll give the
same answer that I gave to Janosch about box2d: what when someone Rock
developer something "like" kdtree ? Where we'll put it ?

Post by Matthias Goldhoorn
I thing we should find a way to handle packages that are purly
downloaded and build by rock (maybe patches), and handle them
separately.

Yes, and I like Leif's answer to that:

Since hopefully the rock community will grow and get more heterogeneous,
the differentiation between internal and external seems obsolete,

You *are* trying to differentiate between "us" and "them". If we want
the community to strive, that difference needs to go away.

Sylvain

Leif Christensen

2014-09-08 09:03:02 UTC

Permalink

Hi,

I agree that the undecided/free-floating state is the maximum annoying one.

In the past I somewhat had the impression, that packages under external/
are really external, meaning untouched by Rock developers and often
directly originate from an external VCS. Than there are manually copied
external SDKs/etc. often nearby own code (camera_prosilica?), sometimes
with slight modifications or glue code.

Most important for me is a coherent directory structure and to see at a
glimpse, if a package has external origins, mainly to get a clou, if the
external code is kept up-to-date.

So I see 3 options:

- keep external/ with flat structure
- add hierarchy to external (external/drivers, external/slam,...)
- move "external" packages to same places as "internal".

Since hopefully the rock community will grow and get more heterogeneous,
the differentiation between internal and external seems obsolete, so I
am going +1 for move "external" packages to same places as "internal".

If that is too much work, I would vote for flat external (since the
number of packages in there is currently not overwhelming and I like
flat hierarchies).

And +2 for announcing, if there has been a decision and how this is
handled from then on...

LG,
Leif

--
Leif Christensen

DFKI Bremen
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Phone: +49 (0)421 17845-4149
Fax: +49 (0)421 17845-4150
E-Mail: leif.christensen at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

Steffen Planthaber

2014-09-08 10:32:10 UTC

Permalink

Hi,

Post by Leif Christensen
Most important for me is a coherent directory structure and to see at a
glimpse, if a package has external origins, mainly to get a clou, if the
external code is kept up-to-date.

+1 "external" folders mean to me "not maintained by rock developers"
(exept making them build with autoproj by patches)

Post by Leif Christensen
- keep external/ with flat structure
- add hierarchy to external (external/drivers, external/slam,...)
- move "external" packages to same places as "internal".

I also see Option 4 (which I prefer):
- keep external/ with packages useful for different topics (kdtree,
boost, etc.) but also have categorized external foldes to help finding
these libs: like /planning/external/ompl

Post by Leif Christensen
Since hopefully the rock community will grow and get more heterogeneous,
the differentiation between internal and external seems obsolete, so I
am going +1 for move "external" packages to same places as "internal".

I think it should be visible in the folder structure if the code in a
package is maintained by the community or by an EXTERNAL community/person

Steffen

Post by Leif Christensen
LG,
Leif

--
Steffen Planthaber
Weltraumrobotik

Besuchsadresse der Nebengesch?ftstelle:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 5
28359 Bremen, Germany

Postadresse der Hauptgesch?ftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Stra?e 1
28359 Bremen, Germany

Tel.: +49 421 178 45-4125
Zentrale: +49 421 178 45-0
Fax: +49 421 178 45-4150 (Faxe bitte namentlich kennzeichnen)
E-Mail: Steffen.Planthaber at dfki.de

Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Stra?e 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------

Sylvain Joyeux

2014-09-08 12:42:17 UTC

Permalink

Post by Steffen Planthaber

+1 "external" folders mean to me "not maintained by rock developers"
(exept making them build with autoproj by patches)

Can you or Matthias explain in practice what is gained by doing this
separation ? How would we deal once let's say a PCL contributor
becomes a Rock developer ? Would we then move pcl to slam/ ? And then
if he leaves we move it back to external ?

In addition, I start to feel that even external packages should have a
"Rock maintainer", which pushes updates / maintains patches / pushes
the patches upstream (ideally). Right now, it's a free-for-all which
is problematic.

Sylvain

Steffen Planthaber

2014-09-08 13:24:49 UTC

Permalink

Hi,

Post by Sylvain Joyeux

Post by Steffen Planthaber
+1 "external" folders mean to me "not maintained by rock developers"
(exept making them build with autoproj by patches)

kind of, yes. When support for a package can be found through the
mailing list, it is not external. The folder structure tells users
directly whether (good) support can be found on the rock-lists, or not
and users of these packages might have to care for themselves. Good
support here means that maintainer can also help by fixing bugs in the
source, not only the inclusion of the source into autoproj.

Post by Sylvain Joyeux
In addition, I start to feel that even external packages should have a
"Rock maintainer", which pushes updates / maintains patches / pushes
the patches upstream (ideally). Right now, it's a free-for-all which
is problematic.

This is exactly the answer to your question. When every package gets a
"Rock maintainer" the external folder will be empty. If such a
maintainer is unavailable, the package goes to external.

Steffen

Post by Sylvain Joyeux
Sylvain

Sylvain Joyeux

2014-09-08 15:13:00 UTC

Permalink

kind of, yes. When support for a package can be found through the mailing
list, it is not external. The folder structure tells users directly whether
(good) support can be found on the rock-lists, or not
and users of these packages might have to care for themselves. Good support
here means that maintainer can also help by fixing bugs in the source, not
only the inclusion of the source into autoproj.

By this logic, most packages should go to external/

Most of our packages are a one-man or two-men development. When there
is a bug, either the guy is around and can fix it or the people can
dream of any support. In addition, if we strive to be more inclusive
(i.e. have more contributors), this will become more and more the
case. As the number of packages go up, we (Rock the project) really
can't guarantee "Good support for each and every one of them". Package
maintainers can, and all the projects we include have maintainers,
just not people that necessarily identify themselves with Rock.

Moreover, this point of view also goes contrary to the experience of
most big projects with dependencies. Users don't care whether it is
"developed by" or "pulled by". They are using Rock, have a problem
with Rock, they are going to ask for help to the Rock developers. And
so should they, we might have broken something ourselves. "This is a
upstream bug" is a perfectly valid answer there, of course, but not
the only possible.

This was one of my point with the release situation: there is no such
thing as a "well supported Rock", only a "well supported Rock core".
The rest is a collection of software developed by various people that
have various priorities, various schedules (like "I can't help you
right now, spacebot is in two weeks", or "I am in the last three
months of my thesis, no time to look into it") and (hopefully) will
come from various places in the world.

It brings me down to the inclusiveness of Rock as a project. I have
the impression that both the point of views on how releases should be
made and external/ boil down to: what should Rock's identity be in the
end.

For the pro-external, I feel that Rock is a showcase to what DFKI
does. Every development is done internally by projects and sometimes
released. Development behind closed doors. Discussions behind closed
doors.

For me (I won't speak for the others), Rock was aimed at being a true
open-source project. Which means that we need to be inclusive to as
much people outside of Rock and *especially* outside DFKI and push and
include devel code so as to show (1) that we are progressing and (2)
favor having people contribute to the development of Rock, not only to
fixing its bugs. That's the only way we can start having a community
and not only DFKI: making decisions in the open, and not separating
"we" and "them".

Sylvain

Jakob Schwendner

2014-09-08 15:48:03 UTC

Permalink

It brings me down to the inclusiveness of Rock as a project. I have the
impression that both the point of views on how releases should be made and
external/ boil down to: what should Rock's identity be in the end.

This seems to be a good interpretation of the situation, and possibly the
clue to resolve it properly.

I think that the external directory should just go away.
Releases imho should be for a set of core packages.
The libraries and components are peripheral and their level of maintenance
will change over time.
Their level of compatibility with releases as well as their maintainership
status could be indicated by tags in the manifest.

Cheers,

Jakob