Stefano Zamboni
2018-01-23 14:31:04 UTC
forwarding this to devinfo
links to the code are still valid/available
HTH
-------- Forwarded Message --------
Subject: Server manager, a new approach (long)
Date: Tue, 7 Feb 2017 00:05:06 +0100
From: Stefano Zamboni <***@mind-at-work.it>
To: Board contribs <***@lists.contribs.org>
Dear mates, friends
During the last (almost) 2 months I worked on a new Server Manager
application.
First of all, let me recap some facts:
- our SM is old, both from a language/technology and graphical point of view
- it's made with a melt of perl and html, which makes it difficult to
understand, develop, expand and doesn't abtract people..
- we have zillions of features/values that we can't set via UI
- we are a small, too small, group of guys and only some of us have perl
skills
- we're working on porting SME to C7, but we can't really and seriously
think to release it with our old SM (even if with a new logo and colors)
- we need to reuse most of the code/logic we already have and stay with
perl
now, a year ago, after FOSDEM, I proposed a new SM based on cockpit.. it
wasn't accepted, for many reasons.. now, many of them are clearer to me.
even if frustrated, I kept thinking about a solution and during my
research, I found patternfly (http://www.patternfly.org) and, more, its
angularjs declination (http://www.patternfly.org/angular-patternfly/#/api)
I then started playing with them and mojolicious.. what I created is
just a prototype, but many things are already working.
at this point I need to clarify some things:
- I'm not a perl developer
- I'm not an angularjs developer
- in my proto I left many things behind (meaning that I did not coded
them) but I spent many hours to research and think about how them are
supposed/expected to work
- all graphics aspects are just there to make my work eyecatching,
nothing is definitive (logo, colours and so on.. Nicolas already knows)
so, going on, here's how I designed it:
- a REST back end created with mojolicious
- an angularjs (v 1.X, the last available on 1 branch), HTML5 frontend
frontend gets/puts data from/to backend using json just calling some
routes created with mojo
some of you already saw it working (and have an old version of it), so
some questions have already been posted.
1) why mojo? well.. it's perl, it has been suggested by Charlie, it's
REST friendly.. it permits to reuse almost all the code we already have
2) why 2 separated entities?
- because we must use rationally our resources.. the perl skilled guys
can work only on the back end side, without even knowing how the UI will
work
- because we can use any language/tecnology to implement one side
without affecting the other one.. the 2 parts just use known APIs to
dialog (see https://www.peplink.com/ic2-api-doc/ to understand what I
mean with known and documented APIs)
- because the API approach opens many unexplorated opportunities.. for
example, I'd connect to many servers (one at time) using only one
interface.. because I'd integrate my server(s) into a bigger plot, just
using known and secure API's
- because I'd create a webpanel without knowing anything about how SME
works behind the scenes.. theorycally speaking, the back end, the html
and the js logic (the angular modules) could be developed by 3 different
teams that uses just some documented and shared knowledge
- because as long as APIs are secure, I can't write anything on the
frontend side that can be dangerous for my server.. I have no direct
access to anything that isn't exposed by the API
3) why angular? because it's OOSS, it's developed and supported by
google, has a big community of developers, it's MVC compliant and REST
friendly, it makes quite easy to create rich and responsive UIs
4) why patternfly? because it's OOSS, it's developed and supported by
RH, it offers an integration with angular
what's in my proto:
- a perl daemon:
you'll find it here
https://mega.nz/#!7U0BWRKD!bRJlO2Bt21vtSsrg4A_otkmZtiKzi3nGAr-OjInBpTM
to setup it, expand the archive in /root (or wherever you prefer),
install the rpms and then start it with
morbo ./daemon.pl
you'd see something like "server available at http://127.0.0.1:3000"
the netdata rpm is not strictly necessary, but it gaves you the
opportunity to see netdata's charts embedded (and added in real time) in
the dashboard
to make netdata work it's necessary to create the symbolic link in
/etc/rc7.d, which lacks in smeserver-netdata rpm (my bad..)
ATM the daemon doesn't use https, but it should be behind an apache
reverse proxy (just like we do with the SM http instance)
this daemon will work on SME9 and 10 as long as we have the rpms to
install the needed modules
if you check the daemon's code, you'll se some generic, rough and
unsecure routes.. they are there just because I need them, but as Dani
said in chat, we must made them secure.. Charlie, Dani, I really
expect/hope you to give us all your perl knowledge here
- the panels
you'll find the app here:
https://mega.nz/#!zdNlHToY!fHkqW-hqNI93VNzx1PnBdIisMzkpdP_HxbKbpmeoy_o
I usually deploy them into Primary's html folder..
to setup them, extract the archive into Primary or create an apache
virtualhost.. it doesn't matter.. the only prerequisite is that the
folder must be "exposed" by apache..
then, from a client of your lan, just surf to http://your_SME_FQDN/ (be
sure SME is the dns server or that its FQDN is correctly resolved..)
you'd see an initial black page that redirects to the login form.. use
admin credentials (but, ATM, you can use any valid user) to login
you'd have something that reminds our SM.. the language might not work,
because of my lack in perl (I need to recover the Accept-Language header
but I was not able to do so)
anyway, the main menu should reproduce the stucture you see into our
usual SM (I use the same navigation db).. I see some issues with french
(and other language ) chars here, but I'm sure it will be a trivial
issue to solve
if you install a contrib, its menu item will be available as well if you
refresh the page
in the right upper corner you'd see the user logged, an utility menu
and, if any update is available or a reconfigure is necessary, an alert
that reminds you about it..
if you don't see anything, just give "config set UnsavedChanges yes" and
wait (the routine runs every 2 minutes)
there are some panels working.. NOTE: I did not created any SAVE feature
(but in the daemon code you'll find a mockup of it)
all the "collaboration" category panels are working.. they retrieve and
show the underlying data.. the translation are used (not everywhere),
and you'd see a question mark icon that shows (opens) a contestualized
help panel
the other panels already done are the "license and support" one and the
reboot/shutdown/reconfigure, which is almost working.. meaning that it
will do exactly what you choose to do :-)
there are so many things I thought about that it's almost impossible for
me to write all them down here..
- security: in REST apps (which are fully "stateless") the best way to
keep trace of logged user is JWT (see
https://github.com/jberger/Mojo-JWT and
https://github.com/auth0/angular-jwt).. due to my lack in perl knowledge
I didn't even tried to play with them.. ATM my proto use a cookie
the angular JWT component will help in enabling/disabling/hiding parts
of the SM using users different from admin
- contribs.. we'd rethink contribs.. ATM, installing a contrib will
create a menu item and a web page with it's logic buried inside.. this
means that, given the fact that FM is crap, if I install
smeserver-password (for example) I'll have almost the same datas (user's
table) in two different pages.. my idea is to have, keeping
smeserver-password as an example, a panel under security which we'll use
to setup general parametes (password strenght etc) and an "extension" in
the useraccounts page.. IOW, if I need to edit an user property, I must
find it in the useraccounts page.. this is achievable with my solution
(I already have some pieces of code)
this solution can live on the same server with the old SM, which is a
good thing for some aspects
the really big (and still unresolved) issue is that.. no, it won't work
with elinks.. no way (AFAIK).. but it's available and fully usable even
from a small smartphone screen.. it's fast and can manage big amount of
data (users, for example) easily.. in my VM I have 220 users, that in
our old SM is a number quite big that tend to make it unusable
I added John and HSF in CC because I don't know if they're still in the
board ML but I think we have to be as many as we can.
guys, I tried to do my best but still willing to learn.. I hope you'll
try my work and ask me as many questions as you can.. I'll post my code
on BZ and start a 3ad in devinfo ML if I'll get some good and
propositive feedback..
finally..
@Charlie, I know this is not the way you proposed mojo and the way you
expect the new SM will work, but if we keep the same approach (even if
with a new tool loke mojo) we used till now, we won't move on..
embedding html into perl makes people stay away from us and move to
something different..
@Steph.. this is something new that nobody has outside here, I hope
it'll have your attention and interest.. the experience you've done with
NS could be useful for us
@Greg: we'd have S4 and the new SM in SME10.. ATM, concentrate yourself
on the logic side and foget about the UI.. once you're done, I'll start
to work on the new panel
@all: please, let's start to collaborate.. if I did it in less than 2
months, I'm sure we can do it in a relatively short time
thank you all for your time, patience and attention, and forgive my
english :-)
Ciao
S.
links to the code are still valid/available
HTH
-------- Forwarded Message --------
Subject: Server manager, a new approach (long)
Date: Tue, 7 Feb 2017 00:05:06 +0100
From: Stefano Zamboni <***@mind-at-work.it>
To: Board contribs <***@lists.contribs.org>
Dear mates, friends
During the last (almost) 2 months I worked on a new Server Manager
application.
First of all, let me recap some facts:
- our SM is old, both from a language/technology and graphical point of view
- it's made with a melt of perl and html, which makes it difficult to
understand, develop, expand and doesn't abtract people..
- we have zillions of features/values that we can't set via UI
- we are a small, too small, group of guys and only some of us have perl
skills
- we're working on porting SME to C7, but we can't really and seriously
think to release it with our old SM (even if with a new logo and colors)
- we need to reuse most of the code/logic we already have and stay with
perl
now, a year ago, after FOSDEM, I proposed a new SM based on cockpit.. it
wasn't accepted, for many reasons.. now, many of them are clearer to me.
even if frustrated, I kept thinking about a solution and during my
research, I found patternfly (http://www.patternfly.org) and, more, its
angularjs declination (http://www.patternfly.org/angular-patternfly/#/api)
I then started playing with them and mojolicious.. what I created is
just a prototype, but many things are already working.
at this point I need to clarify some things:
- I'm not a perl developer
- I'm not an angularjs developer
- in my proto I left many things behind (meaning that I did not coded
them) but I spent many hours to research and think about how them are
supposed/expected to work
- all graphics aspects are just there to make my work eyecatching,
nothing is definitive (logo, colours and so on.. Nicolas already knows)
so, going on, here's how I designed it:
- a REST back end created with mojolicious
- an angularjs (v 1.X, the last available on 1 branch), HTML5 frontend
frontend gets/puts data from/to backend using json just calling some
routes created with mojo
some of you already saw it working (and have an old version of it), so
some questions have already been posted.
1) why mojo? well.. it's perl, it has been suggested by Charlie, it's
REST friendly.. it permits to reuse almost all the code we already have
2) why 2 separated entities?
- because we must use rationally our resources.. the perl skilled guys
can work only on the back end side, without even knowing how the UI will
work
- because we can use any language/tecnology to implement one side
without affecting the other one.. the 2 parts just use known APIs to
dialog (see https://www.peplink.com/ic2-api-doc/ to understand what I
mean with known and documented APIs)
- because the API approach opens many unexplorated opportunities.. for
example, I'd connect to many servers (one at time) using only one
interface.. because I'd integrate my server(s) into a bigger plot, just
using known and secure API's
- because I'd create a webpanel without knowing anything about how SME
works behind the scenes.. theorycally speaking, the back end, the html
and the js logic (the angular modules) could be developed by 3 different
teams that uses just some documented and shared knowledge
- because as long as APIs are secure, I can't write anything on the
frontend side that can be dangerous for my server.. I have no direct
access to anything that isn't exposed by the API
3) why angular? because it's OOSS, it's developed and supported by
google, has a big community of developers, it's MVC compliant and REST
friendly, it makes quite easy to create rich and responsive UIs
4) why patternfly? because it's OOSS, it's developed and supported by
RH, it offers an integration with angular
what's in my proto:
- a perl daemon:
you'll find it here
https://mega.nz/#!7U0BWRKD!bRJlO2Bt21vtSsrg4A_otkmZtiKzi3nGAr-OjInBpTM
to setup it, expand the archive in /root (or wherever you prefer),
install the rpms and then start it with
morbo ./daemon.pl
you'd see something like "server available at http://127.0.0.1:3000"
the netdata rpm is not strictly necessary, but it gaves you the
opportunity to see netdata's charts embedded (and added in real time) in
the dashboard
to make netdata work it's necessary to create the symbolic link in
/etc/rc7.d, which lacks in smeserver-netdata rpm (my bad..)
ATM the daemon doesn't use https, but it should be behind an apache
reverse proxy (just like we do with the SM http instance)
this daemon will work on SME9 and 10 as long as we have the rpms to
install the needed modules
if you check the daemon's code, you'll se some generic, rough and
unsecure routes.. they are there just because I need them, but as Dani
said in chat, we must made them secure.. Charlie, Dani, I really
expect/hope you to give us all your perl knowledge here
- the panels
you'll find the app here:
https://mega.nz/#!zdNlHToY!fHkqW-hqNI93VNzx1PnBdIisMzkpdP_HxbKbpmeoy_o
I usually deploy them into Primary's html folder..
to setup them, extract the archive into Primary or create an apache
virtualhost.. it doesn't matter.. the only prerequisite is that the
folder must be "exposed" by apache..
then, from a client of your lan, just surf to http://your_SME_FQDN/ (be
sure SME is the dns server or that its FQDN is correctly resolved..)
you'd see an initial black page that redirects to the login form.. use
admin credentials (but, ATM, you can use any valid user) to login
you'd have something that reminds our SM.. the language might not work,
because of my lack in perl (I need to recover the Accept-Language header
but I was not able to do so)
anyway, the main menu should reproduce the stucture you see into our
usual SM (I use the same navigation db).. I see some issues with french
(and other language ) chars here, but I'm sure it will be a trivial
issue to solve
if you install a contrib, its menu item will be available as well if you
refresh the page
in the right upper corner you'd see the user logged, an utility menu
and, if any update is available or a reconfigure is necessary, an alert
that reminds you about it..
if you don't see anything, just give "config set UnsavedChanges yes" and
wait (the routine runs every 2 minutes)
there are some panels working.. NOTE: I did not created any SAVE feature
(but in the daemon code you'll find a mockup of it)
all the "collaboration" category panels are working.. they retrieve and
show the underlying data.. the translation are used (not everywhere),
and you'd see a question mark icon that shows (opens) a contestualized
help panel
the other panels already done are the "license and support" one and the
reboot/shutdown/reconfigure, which is almost working.. meaning that it
will do exactly what you choose to do :-)
there are so many things I thought about that it's almost impossible for
me to write all them down here..
- security: in REST apps (which are fully "stateless") the best way to
keep trace of logged user is JWT (see
https://github.com/jberger/Mojo-JWT and
https://github.com/auth0/angular-jwt).. due to my lack in perl knowledge
I didn't even tried to play with them.. ATM my proto use a cookie
the angular JWT component will help in enabling/disabling/hiding parts
of the SM using users different from admin
- contribs.. we'd rethink contribs.. ATM, installing a contrib will
create a menu item and a web page with it's logic buried inside.. this
means that, given the fact that FM is crap, if I install
smeserver-password (for example) I'll have almost the same datas (user's
table) in two different pages.. my idea is to have, keeping
smeserver-password as an example, a panel under security which we'll use
to setup general parametes (password strenght etc) and an "extension" in
the useraccounts page.. IOW, if I need to edit an user property, I must
find it in the useraccounts page.. this is achievable with my solution
(I already have some pieces of code)
this solution can live on the same server with the old SM, which is a
good thing for some aspects
the really big (and still unresolved) issue is that.. no, it won't work
with elinks.. no way (AFAIK).. but it's available and fully usable even
from a small smartphone screen.. it's fast and can manage big amount of
data (users, for example) easily.. in my VM I have 220 users, that in
our old SM is a number quite big that tend to make it unusable
I added John and HSF in CC because I don't know if they're still in the
board ML but I think we have to be as many as we can.
guys, I tried to do my best but still willing to learn.. I hope you'll
try my work and ask me as many questions as you can.. I'll post my code
on BZ and start a 3ad in devinfo ML if I'll get some good and
propositive feedback..
finally..
@Charlie, I know this is not the way you proposed mojo and the way you
expect the new SM will work, but if we keep the same approach (even if
with a new tool loke mojo) we used till now, we won't move on..
embedding html into perl makes people stay away from us and move to
something different..
@Steph.. this is something new that nobody has outside here, I hope
it'll have your attention and interest.. the experience you've done with
NS could be useful for us
@Greg: we'd have S4 and the new SM in SME10.. ATM, concentrate yourself
on the logic side and foget about the UI.. once you're done, I'll start
to work on the new panel
@all: please, let's start to collaborate.. if I did it in less than 2
months, I'm sure we can do it in a relatively short time
thank you all for your time, patience and attention, and forgive my
english :-)
Ciao
S.